Visualization of the output of Sound Event Detection algorithms in Freesound

Master in Sound and Music Computing Universitat Pompeu Fabra Visualization of the output of Sound Event Detection algorithms in Freesound Joaquim Marcé Forns Supervisor: Frederic Font July 2025 Contents 1 Introduction 1 1.1 Motivation.................................. 1 1.2 Objectives.................................. 2 1.3 Structure of the Report . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 State of the art 4 2.1 Foundations of Sound Event detection . . . . . . . . . . . . . . . . . . 4 2.1.1 Problemdefinitions............................. 4 2.1.2 Modelarchitectures............................. 6 2.1.3 Evaluationmetrics ............................. 7 2.2 Datasets and benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 DCASEDatasets.............................. 8 2.2.2 FSD50K................................... 9 2.3 Post-processing of Sound Event Detection outputs . . . . . . . . . . . . 11 2.3.1 Generaltechniques ............................. 11 2.3.2 Learning based post-processing . . . . . . . . . . . . . . . . . . . . . . 12 2.3.3 In the context of Freesound . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4 User interfaces for sound annotations . . . . . . . . . . . . . . . . . . . 14 2.4.1 General visualization techniques . . . . . . . . . . . . . . . . . . . . . . 14 2.4.2 SED systems user interfaces . . . . . . . . . . . . . . . . . . . . . . . . 16 2.4.3 Freesound interfaces and tools . . . . . . . . . . . . . . . . . . . . . . . 16 3 Methods 19 3.1 Postprocessing proposals . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1.1 Empirical observation on FSD50K dataset . . . . . . . . . . . . . . . . 20 3.1.2 Tag recommendation methods applied to SED techniques . . . . . . . . 22 3.1.3 Hierarchy filtering for human-readable results . . . . . . . . . . . . . . 23 3.1.4 Future optimization proposals . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 Visualization of SED in Freesound . . . . . . . . . . . . . . . . . . . . 27 3.2.1 General considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.2 Visualizationdrafts............................. 29 3.2.3 Bestuserinterfaces............................. 33 3.3 User satisfaction experiment design . . . . . . . . . . . . . . . . . . . . 35 3.3.1 DatasetFiles ................................ 36 3.3.2 Experimentdesign ............................. 37 4 Results 39 4.1 Postprocessing approaches . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.2 Qualitative insights on detections . . . . . . . . . . . . . . . . . . . . . 40 4.3 User Satisfaction Survey Results . . . . . . . . . . . . . . . . . . . . . . 41 4.3.1 Annotationaccuracy ............................ 41 4.3.2 Visualization effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.3.3 Qualitative Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5 Discussion 48 5.1 Postprocessing insights . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.2 Visualization Considerations . . . . . . . . . . . . . . . . . . . . . . . . 49 5.3 Futurework................................. 49 5.4 Conclusions ................................. 50 List of Figures 52 List of Tables 55 Bibliography 56 A Resources and Code availability 58 B Filtering results 59 Dedication I would like to dedicate this work to birds. Acknowledgement I would like to express my sincere gratitude to: •My supervisor, Frederic Font, for his utmost patience, unlimited curiosity and detailed attention and support for the past months. •My family, for always being supportive. Chapter 2 State of the art This chapter reviews the fundamentals and latest advancements in the literature of this work. Given that the scope of this thesis involves two main topics, postprocessing for the outputs of sound event detection algorithms, and its visualization and user interface design, they are treated separately. 2.1 Foundations of Sound Event detection 2.1.1 Problem definitions Sound event detection (SED) is a rapidly evolving subfield of computational analysis of sound scenes that involves the automatic identification and classification of acoustic events in real world audio recordings. Sound event detection aims not only at recognizing the presence of sound events; sounds produced by a single source (dog barking, car passing by...) but also at localizing these events in the temporal domain. This subsection introduces the core problem formulations involved in the process of sound event detection: audio tagging, event detection and segmentation and the distinction between monophonic vs polyphonic detections. 4 2.1. Foundations of Sound Event detection 5 Audio Tagging vs Event Detection. Sound scene analysis research is fundamentally divided into audio tagging and event detection. On the one hand, audio tagging refers to the task of assigning one or more labels to an audio clip to determine which classes are present, without specifying its temporal information. This is usually treated as a multi-label classification task (cita al llibre!) and is appropriate for weakly labeled datasets where annotations are provided for each audio clip. Common benchmarks such as Audioset [1] and FSD50K [2] provide large-scale data for this goal. On the other hand, sound event detection requires, not only identifying which classes are present in an audio clip but also delimiting its time boundaries i.e. predicting their onsets and offset along the temporal domain for each event. This requirement transforms the problem into a localization task evaluated using event-based metrics such as F1-Score and error rate (ER) [3]. Therefore, event detection is a more demanding and complex task than tagging, especially when dealing with real-word auditory scenes, where multiple events tend to overlap. Monophonic vs Polyphonic Detections Another division in sound event detection refers to the number of sounds present at a time for an audio clip. In monophonic environments, only one sound event is meant to occur at a time, simplifying the modeling process. However, as mentioned above, many real-world recordings contain overlapping sounds. One just needs to think of a domestic environment such as a kitchen, where is practically impossible for sound events not to overlap (cooking, speech, home appliances etc.). Polyphonic sound event detection addresses this complexity by assuming the overlap of multiple sound events temporal and frequency-wise [4]. This task is particularly important for datasets like DESED and FSD50K, which are relevant to this research and will be inspected in more detail further in this thesis. 6Chapter 2. State of the art Weakly vs Strongly Labeled Data The availability of annotation-quality datasets tends to influence the modeling strategy chosen in each case. These annotations can be separated into weak and strong labels. Strong labels include precise timestamps for each sound event, which facilitates supervised learning for temporal localization, but their main drawback is that they are quite difficult to obtain. By contrast, weak labels only contain cliplevel information regarding the present classes in each clip, a feature that makes them much easier to collect but at the same time implies a significant challenge for learning quality temporal patterns. Recent advances in Multiple instance Learning (MIL) and attention mechanisms have attempted to fill this gap [5]. 2.1.2 Model architectures The performance of SED systems is highly dependent on the model architectures they use. For the recent years, Convolutional Neural Networks (CNNs) and their recurrent extensions have become a popular approach due to their ability to learn robust audio features and representations from spectrogram inputs. This section provides a brief overview of important models used in SED, specifically focusing on the FSD-SINet model, which is crucial to the Freesound analysis pipeline. Convolutional Neural Networks and CRNNs CNNs form the main foundation of many SED systems by combining local timefrequency feature extraction with global feature integration. Their capacity for translation invariance enables them to detect important audio characteristics independently of slight temporal or spectral shifts [4]. To better model temporal dependencies across frames, CNNs are usually combined with Recurrent Neural Networks, resulting in Convolutional Recurrent Neural Networks (CRNNs) that can capture a better context [6]. CRNNs have demonstrated strong performance in polyphonic SED benchmarks such as DCASE challenges. 2.1. Foundations of Sound Event detection 7 The FSD-SINet Model: Increasing Shift-Invariance The FSD-SINet model developed by Fonseca and Serra [7] represents a notorious improvement for the Freesound analysis pipeline. It addresses imperfect shiftinvariance, which is the main limitation for standard CNNs in SED. Standard convolutions can be sensitive to small shifts in the input signal, causing detection issues. However, FSD-SINet integrates Shift-Invariant Convolutions (SINets) that increase the conventional CNN layers to improve robustness to problematic shifts, stabilizing the feature extraction process. The resulting predictions for this model provide improved generalization for the weakly-labeled dataset FSD50K. 2.1.3 Evaluation metrics Evaluating SED systesm requires choosing metrics that can properly evaluate both sound event classification and the detection of the event temporal boundaries. F1 Score The F1 Score is still one of the most popular metrics used for SED evaluation. It represents the harmonic mean of precision and recall, therefore balancing and penalizing both false positives and false negatives. The two most frequently used variants are: •F1-Macro: This metric computes the F1 score independently for each class and averages them, so it gives the same importance to all classes regardless of their frequency in the dataset. •Per-class F1: This metric calculates the F1 score for each class, so that strengths and weaknesses of the model are available for each category to facilitate targeting improvements. F1-based metrics usually treat sounds events as binary occurrences in fixed segments length, which may not take into account temporal localization accuracy. 8Chapter 2. State of the art Event-based metrics Apart from segment-level metrics such as the mentioned F1 Scores, SED evaluation systems often use event-based metrics for the sake of temporal accuracy. These metrics consider time boundaries of the predicted events compared to ground truth annotations [8]. •Event-based F1 Score and ER: Proposed in the DCASE challenges, these metrics evaluate the accuracy of detections based on onset and offsets, handling missed, false and overlapping events. •Polyphonic Sound Detection Score: This metric has been recently introduce to provide a better handling of polyphonic and overlapping events. PSDS integrates both segment-level and event-level detections offering an increase in robustness for real-world applications. Despite PSDS providing richer evaluation results, the F1-Macro and per-class F1 still remain standard and popular choices for benchmarking and comparative analysis in many SED systems studies. Mean Average Precision mAP is a widely used clip-based metric in mutli-label classifications and detections tasks. It measures the average precision (AP) across all classes an it is especially helpful to evaluate how well a model ranks classes over entire audio clips, which is a common practice in weakly labeled sound event detections tasks [7]. 2.2 Datasets and benchmarks 2.2.1 DCASE Datasets The development and benchmarking of Sound Event Detections systems, similarly to other related fields of study, rely on standardized datasets that can provide annotated audio for both training and evaluation. Over the past decade, the SED 2.2. Datasets and benchmarks 9 community has converged around a set of datasets that contain diverse acoustic environments, scenes and events, framed within the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge. DCASE Challenge Datasets The DCASE series, held annually since 2013, has played a fundamental role in common benchmarks for SED and other related tasks. These challenges have introduced several datasets focused on different aspects of audio scene analysis such as audio scene classification, single and multi-channel event detection and weakly supervised learning. [8]. A particularly influential subset is DCASE Task 4, which treats the polyphonic sound event detection under weakly labeled (or unlabeled) conditions being a good approach to deal with real-world situations and constraints. In the context of DCASE Task 4, the DESED (Domestic Environment Sound Event Detection) dataset has been notably important. This dataset has been developed to simulate realistic domestic acoustic environments while remaining controllable and reproducible. DESED contains real and synthetic recordings organized as seen in 1: •The synthetic subset is generated by mixing isolated sound events diverse soundscapes. It provides strong temporal annotations, make it a really good fit for supervised learning to evaluate temporal precision. •The real subset includes unlabeled, weakly, and strongly labeled data separated into training and validation for the development set and the public evaluation set. This dual composition allows to test models under different learning conditions and has become a standard benchmark for evaluating models. 2.2.2 FSD50K FSD50K is a large-scale open dataset designed for multi-label sound classification systems and audio tagging, with a strong emphasis on diversity, realism and scalability. Released in 2020 by Fonseca et al., it is extracted from the Freesound platform 10 Chapter 2. State of the art Figure 1: DESED dataset composition overview and can be used as a critical benchmark for audio classification models training with real-world sound content [2]. Dataset composition FSD50K consists of over 51.000 audio clips extracted from Freesound. These clips are labeled using a subset of the AudioSet ontology [1], using 200 unbalanced sound classes ranging through several labels classified by sound source (human voice) or its means of production (bang), which can be hierarchically sorted according its ontology. The dataset is split into development and evaluation, with weak labels, metadata, and clips with lengths ranging from 0.3 to 30 seconds, reflecting the variability and unpredictability of real-words acoustic scenarios. It is important to highlight that all audio files are drawn from an open domain, and that FSD50K includes an annotator bounded tool to validate category annotations with over 600 contributors. 2.3. Post-processing of Sound Event Detection outputs 11 2.3 Post-processing of Sound Event Detection outputs 2.3.1 General techniques Post-processing plays a crucial role in transforming raw outputs from SED models into human-readable data so that the event detections can be properly interpreted by end-users. This section reviews the most popular techniques such as thresholding and median filtering. Thresholding The most basic yet common method for postprocessing is probability thresholding. Given that SED models tend to output a matrix of frame-wise class probabilities, either a fixed or class-dependent threshold can be applied to determine if the confidence of a present class is enough for the sound event to be active at the given frame. This binarization of the sound event time boundaries converts the model output into a hard decision making process. While it is a straightforward technique, using a global threshold (e.g., 0.5) may underperform for unbalanced sets or when the confidence tends to vary a lot for a single class. For this reason, using class-dependent thresholding is often a reasonable approach, tuning the threshold according to validation data. Median filtering Another popular method for smoothing sound event detection outputs is median filtering. This approach provides a better handling of sudden isolated detections or gaps as seen in 2. Median filtering is applied to the binary predictions over time after thresholding, by using a sliding window of an odd size that typically takes from 3 to 11 frames. The kernel size of the median filter is sometimes determined empirically, depending on the expected behavior of sound classes involved in the process. In some implementations, class-dependent median filtering is used to better match the temporal behavior of the different considered classes. 12 Chapter 2. State of the art Figure 2: Real (green) vs. Predicted (orange) boundaries for an audio event before (up) and after (down) median filtering. Despite these being the two most popular approaches regarding sound event detection postprocessing there are other auxiliary steps such as minimum event duration enforcement or merging temporally adjacent events that can also result helpful. 2.3.2 Learning based post-processing Even though conventional post-processing techniques such as thresholding and median filtering are effective for improving the predictions of SED models, sometimes they still fail to capture the huge complexity of real-world sound scenes. In order to address this limitations, recent research has explored more sophisticated postprocessing methods. A notable example is the approach proposed by Giannakopoulos et al. (2022), in which Reinforcement Learning is used to optimize the postprocessing pipeline of a SED model. In this context, a learning agent learns to configure parameters for the post-processing operations of thresholding and median-filtering to maximize evaluation for metrics such as F1 score or ERR [9]. More specifically, it learns to optimize per-class values for thresholds and median filtering window sizes. The agent is trained using Policy Gradient methods to find the aimed optimal configuration. This approach has shown improvements over manual tuning in experiments on DCASE datasets, highlighting the power of adaptive data-driven postprocessing. As the SED field evolves, the use of learning-based models into the 2.3. Post-processing of Sound Event Detection outputs 13 post-processing pipeline shows a promising direction for future work. 2.3.3 In the context of Freesound In the context of the Freesound analysis pipeline, which is responsible for extracting certain properties from sounds using different analyzers and storing the results in the database, the major analyzer regarding SED is FSD-SINet [7], which is why this work focuses on its results for post-processing and later visualization. However, in this field of study we still can find other notorious analyzers and models worth to mention within the current pipeline. •YAMNet Model: a pre-trained deep net that predicts to the 521 audio event classes in the AudioSet ontology. •Bird-NET Analyzer, which has especially popular among the community of ornithologists due to its ability of identifying specific species of birds by their calls and songs [10]. Postprocessing for FSD-SINet analyzer The current postprocessing pipeline for this model is quite straightforward and one of the main areas of improvement targeted by this work. A static threshold is set to a current value of 0.7 which is used to filter the resulting confidences generated by the FSD-SINet model. After this step, time boundaries are defined following the 0.5 hop-size that divides the audio clip frames. By checking both onsets and offsets for each detection, consecutive events are assembled as a single event detection if there is a match between both parameters. As visited in the previous section, this postprocessing approach can be improved by making it class-dependent and including popular methods such as median filtering. It is important to mention that the Freesound database not only contains the resulting sound event detections from this analysis but also the raw data for the top-10 detected classes for each analyzed audio clip. 20 Chapter 3. Methods 3.1.1 Empirical observation on FSD50K dataset Regarding the use of a static threshold for the hard decision-making process on the output probabilities, the usage of specific, per-class thresholds arises. Given the diversity in both the Freesound audio collection and the set of classes present in the FSD50K dataset, it is evident that an adaptive threshold will improve the global performance of the analyzer. For example, intuitively the singularity of a sound event labeled as applause requires a less strict threshold than more ambiguous labels such as boiling or gurgling. Apart from obtaining an adaptive threshold per class, the lack of a median filtering step in the post-processing raises what could be done in this aspect too. Following the previous idea, setting an specific window size per class helps out smoothing output detections, given that in the temporal domain there is also a wide range of behaviors depending on class (see Thunder and Thunderstorm). In order to obtain adaptive parameters for both operations, a simple observational approach is proposed. It consists of taking a set of different kernel sizes and computing thresholds with different restriction parameters per class and audio clip. By postprocessing the evaluation set with each group of variables and comparing them to ground-truth annotations, the best threshold and kernel sizes can be selected for each class. Threshold computations To determine class-specific decision thresholds, a multi-stage computation process is employed. For each raw analysis file and corresponding class, an array of framelevel thresholds is extracted based classes marked as active in the ground truth annotations. Once this per-frame threshold progression is obtained for a given class and audio clip, a segmentation process is applied. Within each segment, the mean and standard deviation of the activation scores are computed and combined to form a set of representative thresholds: 3.1. Postprocessing proposals 21 frameThresholds ={avg(s) + std(s)|s∈segments} A file-level threshold is then computed by averaging these frame-level values: fileThreshold =avg(frameThresholds,weighted =False) At this stage, both weighted and simple averages are considered by the length of the active segments. The outcome of this process is a set of Nfile-level thresholds for each class, where Ndenotes the number of analysis files available in the dataset. To aggregate these into a single, representative threshold per class, a percentile-based reduction is applied across the Nvalues. This enables the derivation of class-specific thresholds with varying degrees of restrictiveness. Postprocessing subsets At this point of the postprocessing pipeline, a total of 2Pthreshold sets are available, where P denotes the number of percentiles considered in the previous step. Each threshold is computed using both weighted and simple averaging approaches, resulting in two variants per percentile. In order to determine the optimal combination of thresholds and median filter kernel size per class, a binary prediction matrix must be obtained for every file and threshold set. However, the raw activation outputs provided by the analysis files only contain scores for the top 10 detected classes per frame. To reconstruct the complete activation matrix, missing frame-class combinations are assigned a value of 0. Given the reconstructed activation matrix A, predictions are computed for each threshold set T, kernel size k∈ K, and class i∈ C as follows: filePredictionsk,T = MedianFilterk({1[ai> Ti]|i= 1, . . . , C, a ∈ A}),∀k∈ K 22 Chapter 3. Methods This procedure results in 2P K prediction sets per file. In the current configuration, the set of kernel sizes is defined as K= [1,3,5,7] and percentiles considered are P= [50,60,70,80] resulting in 32 different postprocessing configurations. Finally, for each class, the combination of threshold and kernel size that maximizes the F1-score on validation data is selected and recorded in the final postprocessing set. 3.1.2 Tag recommendation methods applied to SED techniques Sound Event Detection (SED) systems operate across a wide variety of sounds, each characterized by different levels of polyphony, ambient conditions and recording equipment. Given this diversity, the per-class parameter selection methods described in the previous section may lack scalability and generalizability, especially when applied to large-scale, heterogeneous audio collections such as Freesound. To address this limitation, it becomes important to think of adaptive strategies that do not rely on other sounds statistical observations. Since the ground truth annotations provide only week labels, the SED task can, in this context, be approached similarly to an audio tagging problem. In this field of study, the tag selection methods for recommendation purposes proposed by Font, 2015 [15] are a solid baseline. Among these, the most effective method described in Section 3.2.3 ("Selection of tags to recommend") is the percentage strategy, which involves selecting tags whose scores surpass a fixed percentage of the highest tag score. Exporting this approach to the current SED task introduces certain considerations. In fact, this tag recommendation method is designed to always output at least one tag, and this assumption does not hold in SED, where many frames can contain no active sound events. Therefore, a class is only considered active at a given frame if its confidence score surpasses both a static threshold (e.g., 0.45) and at least 80% of the highest confidence score observed among all classes for that frame. 3.1. Postprocessing proposals 23 ActiveClass(c, t) =      1if ac,t > τmin and ac,t > α ·max j∈C aj,t 0otherwise Where ac,t is the activation score for class cat frame t,τmin is a fixed minimum threshold (0.45), αis the relative threshold parameter (0.8), and Cis the set of all classes. 3.1.3 Hierarchy filtering for human-readable results The current output format of the FSD-SINet model is structured as a dictionary containing the following elements: •A list of dictionaries with event detection information, including class name, time boundaries, and confidence score. •A list of all unique classes detected within the audio file. •A list of embeddings resulting from the audio analysis. While this structure provides the essential data needed to build a visualization system for Freesound users, two important considerations must be addressed to improve the human readability and interpretability of the output. On the one hand, it is important to acknowledge that the vocabulary used by the FSD50K dataset is derived from a subset of the AudioSet Ontology, which introduces a valuable but currently overlooked detail: class hierarchy. Extracting the hierarchical level for each class can significantly help organize the presentation of sound events during the visualization stage for a more intuitive and informative user experience. Note that, as the ground truth vocabulary does not include some 0-level classes (e.g. Sounds of things all its child classes levels will be pushed up. On the other hand, the current system seeks an integrated display of SED data within the sound players in the Freesound platform, which poses certain limitations 24 Chapter 3. Methods in screen space. To meet this requirement, avoiding redundancy in the information display is crucial. Therefore, when multiple classes that share same time boundaries within a difference of 1 frame (0.5 seconds) are found to have a parent-child relationship in the ontology, they are merged into a single one. In such cases, the child class is selected for display because it provides more specificity for the detected sound event, as seen in 8 and 9. The code with the postprocessing and filtering of FSD-SINet results implementations can be found in appendix A. Figure 8: Draft display for raw detections given by the FSD-SINet model current posprocessing. Figure 9: Draft display for hierarchically filtered detections. 3.1. Postprocessing proposals 25 3.1.4 Future optimization proposals Upon reviewing the previously discussed postprocessing strategies, it becomes straightforward that none of the presented implementations incorporate strong optimization techniques or learning-based approaches to refine postprocessing parameters with the goal of improving evaluation metrics. This highlights a methodological gap considering that optimization-driven methods are more aligned with current research trends in sound event detection. As seen in section 2.3.2, the reinforcement learning (RL) approach proposed by Giannakopoulos et al.(2022)[9] introduces a framework capable of optimizing the entire postprocessing pipeline of a SED model such as FSD-SINet. While promising, this approach presents several challenges that must be addressed. The most notable one is the requirement for strongly labeled annotations, something the FSD50K dataset does not provide. Nevertheless, as mentioned in section 2.2.1, the DESED dataset includes a subset of recordings with strong labels. Although these annotations are limited to domestic sound environments, several of the classes can be mapped to the FSD50K taxonomy. This opens the possibility of training the RL-based optimization system on the strongly labeled DESED subset and exporting the resulting parameters to the corresponding class subset in FSD50K. A proposal for mapping both subsets can be found below, in table 1. Note that the match between classes is challenging due to the presence of ambiguity in resolving specific classes from one set to the other. Table 1: Mapping between DESED classes and FSD50K classes DESED Class FSD-SINet Class Speech (0) Child_speech_and_kid_speaking (33) Female_speech_and_woman_speaking (75) Male_speech_and_man_speaking (111) Continued on next page 26 Chapter 3. Methods Table 1 – continued from previous page DESED_CLASS FSD_SINET_CLASS Speech (158) Human_voice (101) Chatter (29) Conversation (43) Dog (1) Dog (59) Bark (7) Domestic_animals_and_pets (60) Cat (2) Domestic_animals_and_pets (60) Cat (28) Meow (116) Alarm/bell/ringing (3) Alarm (4) Bell (11) Ringtone (140) Bicycle_bell (13) Church_bell (38) Cowbell (45) Doorbell (63) Dishes (4) Dishes_and_pots_and_pans (58) Frying (5) Frying_(food) (83) Blender (6) Domestic_sounds_and_home_sounds (61) Running water (7) Water (187) Water_tap_and_faucet (188) Bathtub_(filling_or_washing) (10) Fill_(with_liquid) (76) Sink_(filling_or_washing) (151) Toilet_flush (175) Vacuum cleaner (8) Domestic_sounds_and_home_sounds (61) Continued on next page 3.2. Visualization of SED in Freesound 27 Table 1 – continued from previous page DESED_CLASS FSD_SINET_CLASS Electric shaver/toothbrush (9) Domestic_sounds_and_home_sounds (61) 3.2 Visualization of SED in Freesound This section presents all procedures regarding the development and implementation of sound event detection displays in Freesound, from general considerations to take into account for all upcoming paradigms, to presenting draft visualizations and implementing the best candidates, and finally designing how the user satisfaction evaluation will be performed. 3.2.1 General considerations This subsection outlines the development and implementation process for displaying sound event detection (SED) results from the FSD-SINet model within the Freesound platform. It begins with general considerations that apply across all subsequent visualization paradigms, followed by the presentation of draft display proposals, the implementation of selected candidates, and the definition of the methodology to evaluate user satisfaction. Data features The visualization of FSD-SINet results for a given audio clip will include the following elements: •The class name of the detected event. •The time boundaries indicating the temporal location of the event. •The confidence score associated with the detection. •The hierarchical level of the class within the ontology. 28 Chapter 3. Methods These features can be grouped based on their relevance for the user interface. On the one hand, class names and time boundaries carry the most critical information and should therefore be prioritized in any visualization approach. On the other hand, the confidence score and hierarchy level, while not crucial, can improve user understanding and interpretation of the detection results. Note that a common property across all displays is that color coding is used for class identification, accompanied by a legend appended below the player container. The chosen set of colors is selected to avoid interference with Freesound default waveform display colors. As a result, the specific format and encoding of the information may vary between different visualization proposals, with each one presenting these parameters using different visual strategies depending on its intended use and emphasis. Finally, it is important to mention that the displayed results correspond to the current postprocessing pipeline for the FSD-SINet model, adding the ontology filter process described in Section 3.1.3 User interaction Similarly to the data features described above, the proposed visualization paradigms also share a set of common user interaction functionalities. These shared properties are designed to ensure intuitive and efficient interaction with detected events. •Mouse click interaction: Clicking on a displayed sound event will automatically move the progress bar to the event’s start time and begin playback from that point. This feature is designed to facilitate user evaluation and to enable quick verification of detected events. •Mouse hover functionality: when hovering over a visualized event, a label will appear showing key details such as the event’s class name, confidence score and time boundaries. This ensures that users can access detailed information without cluttering the main interface. •Overlay display behavior: By default, SED visualizations will not be shown for analyzed sounds. Instead, a toggle button integrated into the audio player’s 3.2. Visualization of SED in Freesound 29 control bar will allow users to show or hide a visual overlay containing the detections information. This layered approach aims to maintain a clean interface while still providing access to the analysis results. These interaction features are particularly relevant given that the proposed visualizations vary in the amount of information displayed by default. Also, they have been designed for enabling user experience with the results in a meaningful and non-intrusive manner. 3.2.2 Visualization drafts This section introduces a series of visualization drafts each offering a distinct approach to presenting sound event detection results. Every proposal is outlined with its key strengths and potential limitations to provide a comprehensive comparison and later choose the best candidates. VisualiSED 01: Class-wise distributed rectangles This approach arranges labels along different vertical levels, each corresponding to a separate sound class. Class names are displayed at the start of the player container in their corresponding level, while individual event detections are represented as rectangles extending over their time intervals as seen in 10. Each rectangle includes a text of the confidence score expressed as a percentage. Whenever space is insufficient for the confidence display due to short sound events, this information is still accessible through mouse hover interaction. A significant strength of this design is its effective handling of overlapping events, which tends to be a challenging task in the visualization of SED outputs. By grouping events by class and allocating each to a different vertical track, overlapping is avoided. However, this approach can have some trouble with sounds containing a high number of distinct classes. VisualiSED 02: Level-wise distributed rectangles To address the space limitations identified in the previous proposal, this method retains the same basic structure (representing events as rectangles indicating their time 36 Chapter 3. Methods 3.3.1 Dataset Files The sounds selected for the final user satisfaction experiment are chosen based on a set of criteria to ensure a representative range of SED behaviors while keeping the experiment efficient. •Dataset size must be kept small (10-20 sounds) in order to make the experiment agile enough. •It must contain a representative sample of various classes present in the FSD50K dataset. •It must contain diverse files in terms of event density and variety (from field recordings to single event audio clips). •Recordings must not be longer than 30 seconds for the analysis to be run without skipping. Based on these considerations, the following sound IDs have been selected from the Freesound development database: 463464, 75825, 347223, 217543, 685989, 682534, 436790, 49520, 671901, 181628, 437623, 463472. For more information of its type of content see the following 2, with classified sounds according to the AudioSet Ontology parent class and the amount of sources present in sound (single vs multiple). Table 2: Experiment dataset ID Source Human Animals Music Ambiguous Things Natural 463464 Multiple True False False False False False 347223 Multiple True False False False False False 217543 Multiple False False True False False False 685989 Multiple False False False False True False 682534 Single False True False False False False Continued on next page 3.3. User satisfaction experiment design 37 Table 2 – continued from previous page ID Source Human Animals Music Ambiguous Things Natural 436790 Multiple False False False False False True 49520 Multiple True True False False False False 671901 Single False False True False False False 181628 Multiple False False False True True False 463472 Multiple True False True False False False 437623 Multiple True False False True True False 75825 Multiple False False False False True True 3.3.2 Experiment design To assess user satisfaction with the selected SED visualization designs, an online mock experiment has been conducted using Google Forms. The form presents participants with previously outlined set of analyzed Freesound sound URLs, containing visualizations generated using the final set of display designs described in the previous section. The main objective of the experiment is to obtain a general assessment and evaluate the clarity, usability, and perceived usefulness of each visualization approach as well as the quality of the generated detections. Participants will be asked to listen to each audio clip while observing the corresponding visual representations and then answer a series of short questions to capture their impressions. The final questionnaire is short and concise to encourage participation. For each visualization design, there is one question addressing the quality of the event annotations while the remaining focus on the effectiveness of the visual design: helpfulness, layout clarity, considering hypothetical usefulness when exploring sounds in Freesound and an optional comment box for any suggestions etc. The 4 sounds visited for each visualization display are randomly selected. The responses have been collected using a 0−10 grading scale and a 5-point Likert scale ranging from Strongly agree to Strongly disagree as follows. 38 Chapter 3. Methods 1. How would you rate the accuracy and quality of the event annotations in this visualization? 0-10 grade scale. 2. The visualization helped me understand the events present in the audio clip. 5-point Likert scale. 3. The layout and visual organization of the events were clear and easy to interpret. 5-point Likert scale. 4. I think this visualization would be useful when exploring sounds in Freesound. 5-point Likert scale. 5. Optional: Any suggestions or observations about this visualization? Text answer. Additionally, at the end of the survey, there is an optional box to leave any message or observations the user wishes to provide. Due to the impact that the proposed changes could have on Freesound, both in terms of visualization displays and the integration of the modifications in the FSDSINet analyzer, the experiment has not yet been distributed to the wider Freesound community. Instead, it has been carried out with a smaller group of participants close to me, including specialists in audio and UX design as well as individuals without prior knowledge of the field. Chapter 4 Results 4.1 Postprocessing approaches The current postprocessing strategy for the FSD-SINet model relies on a fixed threshold of 0.7, which is applied to retain only the most confident detections. In this work, two alternative approaches have been explored: (i) class-dependent thresholds and kernel sizes determined through empirical observation, and (ii) the use of tag-recommendation techniques adapted to sound event detection. For the evaluation of these approaches, the input consisted of the raw outputs of the FSD-SINet model, namely the top-10 frame-wise class confidences for the FSD50K evaluation set. Performance was measured by computing the F-Score for each methodology, with the resulting values summarized in Table 3. In the first method, based on empirical observation, parameters were estimated using the development set. For each class, this process produced a specific percentile, a kernel size, and an indication of whether the averaging should account for detection length. Analysis of these results showed that the most common configuration corresponded to the 50th percentile with kernel size k= 1, without length-weighted averaging. However, this choice of parameters is also the least intrusive, making the approach closely resemble the baseline and therefore reducing its potential impact. 39 40 Chapter 4. Results Considering the F-Score of each postprocessing approach, for the final user experiment the used postprocessing is the current static threshold combined with the hierarchy filtering described in section 3.1.3. 50 60 70 80 0 0.2 0.4 0.6 0.8 1 P values Distribution 1357 0 0.2 0.4 0.6 0.8 1 K values Distribution Figure 16: Distributions for percentiles Pand kernel size values Kparameters. Table 3: Comparison of approaches and their performance Approach Useful parameters Macro F1 Per-class threshold and kernel size Percentiles = [50, 60, 70, 80] Kernel sizes = [1, 3, 5, 7] 0.3385 Tag recommendation percentual approach Min. threshold = 0.45 Valid percentage = 0.8 0.344 Current static threshold Threshold = 0.7 0.3869 4.2 Qualitative insights on detections Given the ultimate goal of this work to evaluate the real-world user experience, it is important to provide insight into how the hierarchy filtering step described in 3.1.3 affects both the display and specificity of detections. The full collection of images can be found in the appendix B, though the most notable effects are illustrated in 17. It is clear that the filtering process not only removes some general detections but also organizes the remaining ones according to a hierarchy, displaying parent-child relationships from top to bottom. 4.3. User Satisfaction Survey Results 41 While the aim of this feature is to improve specificity and enhance usability for the end user, it can sometimes lead to less accurate displays. In particular, a parent class with more accurately defined time boundaries may be omitted in favor of more specific, but less precise, child classes. This effect is illustrated in 17, where the broader Bird class better matches the full duration of the bird singing than the more specific Bird vocalization and bird call and bird song class. 4.3 User Satisfaction Survey Results This section presents the results of the user satisfaction survey, which evaluated both the underlying detection quality of the FSD-SINet model and the effectiveness of three different visualization designs for displaying these detections. A total of 12 participants completed the survey. Each participant listened to four sounds per visualization, randomly selected from the set of 12 sounds, ensuring all sounds were evaluated while introducing variation in the sequence. The questionnaire structure is previously defined in section3.3.2. The results are presented in groups by evaluation topic and all the materials related to the survey can be found in appendix A. 4.3.1 Annotation accuracy Given that all visualization designs rely on the same underlying detections, this assessment directly reflects user confidence in the model outputs. The distribution of responses (Figure 18) shows a central tendency around the middle of the scale (Mean = 4.31), which indicates only moderate perceived accuracy, with noticeable variability (SD = 1.34) and a wide range of opinions. While some users considered the event annotations reasonably accurate, others found them less convincing, suggesting that events were sometimes missed or imprecisely localized. These evaluations highlight the need for further refinement in detection performance and postprocessing approaches to improve user interpretability of SED outputs. In parallel to the survey, a small manually annotated dataset was prepared for the twelve sounds included in the experiment that can be found in appendix A. These annotations provide a reference for assessing the alignment between the FSD- 42 Chapter 4. Results SINet detections and the actual acoustic events. Although a detailed quantitative comparison was outside the scope of this work, preliminary observations suggest that discrepancies between manual annotations and model outputs explain the lower user ratings. This dataset can therefore serve as a valuable baseline for future validation and improvement efforts. 4.3.2 Visualization effectiveness The results of the questions referring to the performance of the different visualization designs reveal differences in how participants perceive the quality of the three displays. Regarding the question of whether the visualization helped users understand the events present in the audio clip (Q2), the Class-Wise design (VS01) received the strongest positive feedback, with 11 participants agreeing and 1 strongly agreeing, and no negative responses. The Detailed Onsets design (VS06) was also positively received, with 8 agreeing and 3 strongly agreeing, though one participant expressed neutrality. The Level-Wise design (VS02) showed more mixed results, with 5 participants agreeing, 3 neutral, and 4 disagreeing, suggesting some users found it less intuitive for interpreting event sequences. Results can be checked in figure 19 A similar pattern emerges for layout clarity and ease of interpretation (Q3), see figure 20. The Class-Wise design again scored highly, with 8 agreeing and 3 strongly agreeing, whereas the Detailed Onsets design had 8 strongly agree responses but fewer agreeing responses overall. The Level-Wise design had more dispersed ratings, including one disagreement and two neutral responses, indicating that hierarchical grouping of events introduces some ambiguity in visual organization. For perceived usefulness when exploring sounds in Freesound (Q4), both the ClassWise and Detailed Onsets designs were consistently rated as useful, with 8 or more participants selecting agree or strongly agree. The Level-Wise design received more varied responses, with some participants neutral or disagreeing, reinforcing the observation that while hierarchical grouping may provide structure, it might reduce interpretability and perceived usefulness in practical exploration of sound events (see figure 21). Overall, these results suggest that the Class-Wise and Detailed On- 4.3. User Satisfaction Survey Results 43 sets designs were generally preferred by users, offering clearer and more actionable visualizations for understanding sound events. 4.3.3 Qualitative Feedback Summarizing the user’s feedback across the three visualization designs, participants generally highlight issues with the quality and accuracy of the detections. Many of them were missing, temporally misaligned, or overly general (e.g., “Musical instrument” for orchestral passages, “Domestic animals” instead of “Dog,” “Human group action” instead of “Applause”). Long events were often truncated, while background sounds such as footsteps or wind were undetected. Several users emphasized that the weak detection quality made it difficult to evaluate the visualizations themselves. VisualiSED 02: Hierarchy Level-Wise Display Users appreciated the use of confidence-dependent borders, but found overlapping rectangles confusing and visually cluttered. Labels missing event names (relying only on colors, with names displayed only on the legend) were seen as problematic for accessibility. Additionally, some users reported that the display of the labels on top of the waveform is confusing, and proposed a separate whole display similar to the toggle between waveform and spectrogram displays. VisualiSED 06: Detailed Onsets Display This design was generally better received. The stacking of overlapping events was considered a clear improvement, making the display less intrusive. However, temporal inconsistencies remained a major issue, particularly for applause and other continuous sounds. Some users criticized redundancy in displaying similar labels (e.g., “Motor Vehicle” multiple times). While still affected by generalistic detections, this display was often described as the most comfortable to use. 44 Chapter 4. Results VisualiSED 01: Class-Wise Display Several users preferred this design because of clearer alignment between detection names and their position (labels listed on the left). However, clutter remained a concern, particularly when many events were present. The overlapping of waveform and labels again drew criticism. Despite persistent detection issues, some considered this an improved version of Design A. General Feedback Overall, users saw potential in the visualizations but stressed that the system is not yet ready for Freesound integration. Suggestions included: separating waveforms from labels to improve readability and allowing toggling of detections via the legend to address color-blind accessibility. While detection quality was the main limitation, participants agreed that visualization could become a useful feature, especially for browsing longer audio files where it could save listening time. 4.3. User Satisfaction Survey Results 45 Figure 17: Top: Sound 682534 detections before filtering. Bottom: Sound 682534 filtered detections. Both displays following the visualization technique described in 3.2.3 List of Figures 1 DESED dataset composition overview . . . . . . . . . . . . . . . . . . 10 2 Real (green) vs. Predicted (orange) boundaries for an audio event before (up) and after (down) median filtering. . . . . . . . . . . . . . 12 3 Soundcloud comments displayed along time axis each at a time instant. 14 4 Different types of sound annotations generated by Sonic Visualizer, each on a different label. Note that each annotation can have an associated customizable text. Text (blue), notes (red), regions (green), boxes (purple), time instants (bright red, vertical line). . . . . . . . . 15 5 Left: BirdNET App in the analysis boundaries setting step. Right: BirdNET App displaying detection results. . . . . . . . . . . . . . . . 17 6 Freesound display in waveform mode. . . . . . . . . . . . . . . . . . . 18 7 Freesound display in spectrogram mode. . . . . . . . . . . . . . . . . 18 8 Draft display for raw detections given by the FSD-SINet model currentposprocessing. ............................ 24 9 Draft display for hierarchically filtered detections. . . . . . . . . . . . 24 10 Draft display for VisualSED 01 . . . . . . . . . . . . . . . . . . . . . 30 11 Draft display for VisualiSED 02 . . . . . . . . . . . . . . . . . . . . . 30 12 Draft display for VisualiSED 03 . . . . . . . . . . . . . . . . . . . . . 31 13 Draft display for VisualiSED 04 . . . . . . . . . . . . . . . . . . . . . 32 14 Draft display for VisualiSED 05 . . . . . . . . . . . . . . . . . . . . . 33 15 Draft display for VisualiSED 06 . . . . . . . . . . . . . . . . . . . . . 34 16 Distributions for percentiles Pand kernel size values Kparameters. . 40 52 LIST OF FIGURES 53 17 Top: Sound 682534 detections before filtering. Bottom: Sound 682534 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 45 18 User answers regarding quality and accuracy of detections. Mean score = 4.31, StdDev = 1.34, Range = 6. . . . . . . . . . . . . . . . . 46 19 User answers regarding helpfulness of the display. . . . . . . . . . . . 46 20 User answers regarding clarity of the display. . . . . . . . . . . . . . . 47 21 User answers regarding usefulness of the display. . . . . . . . . . . . . 47 22 Top: Sound 49520 detections before filtering. Bottom: Sound 49520 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 60 23 Top: Sound 75825 detections before filtering. Bottom: Sound 75825 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 61 24 Top: Sound 181628 detections before filtering. Bottom: Sound 181628 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 62 25 Top: Sound 217543 detections before filtering. Bottom: Sound 217543 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 63 26 Top: Sound 347223 detections before filtering. Bottom: Sound 347223 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 64 27 Top: Sound 436790 detections before filtering. Bottom: Sound 436790 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 65 28 Top: Sound 437623 detections before filtering. Bottom: Sound 437623 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 66 54 LIST OF FIGURES 29 Top: Sound 463464 detections before filtering. Bottom: Sound 463464 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 67 30 Top: Sound 463472 detections before filtering. Bottom: Sound 463472 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 68 31 Top: Sound 671901 detections before filtering. Bottom: Sound 671901 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 69 32 Top: Sound 685989 detections before filtering. Bottom: Sound 685989 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 70 33 Top: Sound 685989 detections before filtering. Bottom: Sound 685989 filtered detections. Both displays following the visualization techniquedescribedin3.2.3.......................... 71 List of Tables 1 Mapping between DESED classes and FSD50K classes . . . . . . . . 25 2 Experimentdataset............................ 36 3 Comparison of approaches and their performance . . . . . . . . . . . 40 55 Bibliography [1] Gemmeke, J. F. et al. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 776–780 (2017). [2] Fonseca, E., Favory, X., Pons, J., Font, F. & Serra, X. Fsd50k: An open dataset of human-labeled sound events (2022). URL https://arxiv.org/abs/2010. 00475.2010.00475. [3] Mesaros, A., Heittola, T., Virtanen, T. & Plumbley, M. D. Sound event detection: A tutorial. IEEE Signal Processing Magazine 38, 67–83 (2021). [4] Adavanne, S., Pertilä, P. & Virtanen, T. Sound event detection using spatial features and convolutional recurrent neural network (2017). URL https:// arxiv.org/abs/1706.02291.1706.02291. [5] Kong, Q., Xu, Y., Sobieraj, I., Wang, W. & Plumbley, M. D. Sound event detection and time–frequency segmentation from weakly labelled data. IEEE/ACM Transactions on Audio, Speech, and Language Processing 27, 777–787 (2019). URL http://dx.doi.org/10.1109/TASLP.2019.2895254. [6] Cakir, E., Parascandolo, G., Heittola, T., Huttunen, H. & Virtanen, T. Convolutional recurrent neural networks for polyphonic sound event detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25, 1291–1303 (2017). URL http://dx.doi.org/10.1109/TASLP.2017.2690575. 56 BIBLIOGRAPHY 57 [7] Fonseca, E., Ferraro, A. & Serra, X. Improving sound event classification by increasing shift invariance in convolutional neural networks (2021). URL https: //arxiv.org/abs/2107.00623.2107.00623. [8] Mesaros, A., Heittola, T. & Virtanen, T. Metrics for polyphonic sound event detection. Applied Sciences 6, 162 (2016). [9] Giannakopoulos, P., Pikrakis, A. & Cotronis, Y. Improving post-processing of audio event detectors using reinforcement learning. IEEE Access 10, 84398– 84404 (2022). [10] Kahl, S., Wood, C. M., Eibl, M. & Klinck, H. Birdnet: A deep learning solution for avian diversity monitoring. Ecological Informatics 61, 101236 (2021). [11] SoundCloud Help Center. Commenting basics (2025). URL https://help. soundcloud.com/hc/en-us/articles/115003566008-Commenting-basics. Accessed: 2025-07-01. [12] Cannam, C., Landone, C. & Sandler, M. Sonic visualiser: an open source application for viewing, analysing, and annotating music audio files. 1467–1468 (2010). [13] Boersma, P. & Weenink, D. Praat, a system for doing phonetics by computer. Glot international 5, 341–345 (2001). [14] Cornell Lab of Ornithology and Chemnitz University of Technology. Birdnet: Bird sound identification. https://play.google.com/store/apps/details? id=de.tu_chemnitz.mi.kahst.birdnet (2025). Mobile application; Android; updated 2025-06-12. [15] Corbera, F. F. Tag Recommendation using Folksonomy Information for Online Sound Sharing Platforms. Ph.d. dissertation, Universitat Pompeu Fabra, Barcelona, Spain (2015). URL https://www.tdx.cat/handle/10803/296797. Appendix A Resources and Code availability Freesound repository branch containing the main code of the project regarding visualizations and UI/UX. Freesound audio analyzers repository branch containing the code for the post-processing and filtering of the outputs of the FSD-SINet model. Dataset useful information and manual annotations in Google Sheets format. User experiment form in Google Forms format. User experiment results collected in a Google Sheets document. 58 Appendix B Filtering results 59 60 Appendix B. Filtering results Figure 22: Top: Sound 49520 detections before filtering. Bottom: Sound 49520 filtered detections. Both displays following the visualization technique described in 3.2.3 61 Figure 23: Top: Sound 75825 detections before filtering. Bottom: Sound 75825 filtered detections. Both displays following the visualization technique described in 3.2.3 68 Appendix B. Filtering results Figure 30: Top: Sound 463472 detections before filtering. Bottom: Sound 463472 filtered detections. Both displays following the visualization technique described in 3.2.3 69 Figure 31: Top: Sound 671901 detections before filtering. Bottom: Sound 671901 filtered detections. Both displays following the visualization technique described in 3.2.3 70 Appendix B. Filtering results Figure 32: Top: Sound 685989 detections before filtering. Bottom: Sound 685989 filtered detections. Both displays following the visualization technique described in 3.2.3 71 Figure 33: Top: Sound 685989 detections before filtering. Bottom: Sound 685989 filtered detections. Both displays following the visualization technique described in 3.2.3