Leveraging Unlabelled Data in Multiple-Instance Learning Problems for Improved Detection of Parkinsonian Tremor in Free-Living Conditions

1 Leveraging unlabelled data in multiple-instance learning problems for improved detection of Parkinsonian tremor in free-living conditions Alexandros Papadopoulos, and Anastasios Delopoulos, Member, IEEE Abstract—Data-driven approaches for remote detection of Parkinson’s Disease and its motor symptoms have proliferated in recent years, owing to the potential clinical benefits of early diagnosis. The holy grail of such approaches is the free-living scenario, in which data are collected continuously and unobtrusively during every day life. However, obtaining fine-grained ground-truth and remaining unobtrusive is a contradiction and therefore, the problem is usually addressed via multiple-instance learning. Yet for large scale studies, obtaining even the necessary coarse ground-truth is not trivial, as a complete neurological evaluation is required. In contrast, large scale collection of data without any ground-truth is much easier. Nevertheless, utilizing unlabelled data in a multiple-instance setting is not straightforward, as the topic has received very little research attention. Here we try to fill this gap by introducing a new method for combining semi-supervised with multipleinstance learning. Our approach builds on the Virtual Adversarial Training principle, a state-of-the-art approach for regular semi-supervised learning, which we adapt and modify appropriately for the multiple-instance setting. We first establish the validity of the proposed approach through proof-of-concept experiments on synthetic problems generated from two well-known benchmark datasets. We then move on to the actual task of detecting PD tremor from hand acceleration signals collected in-the-wild, but in the presence of additional completely unlabelled data. We show that by leveraging the unlabelled data of 454 subjects we can achieve large performance gains (up to 9% increase in F1-score) in per-subject tremor detection for a cohort of 45 subjects with known tremor ground-truth. In doing so, we confirm the validity of our approach on a real-world problem where the need for semi-supervised and multipleinstance learning arises naturally. Index Terms—Deep learning, Semi-supervised learning, Multiple-instance learning, Parkinson’s Disease, Disease monitoring I. INTRODUCTION PARKINSON’S disease (PD) is a long-term neurological disorder associated with both motor and non-motor symptoms, like extremity tremor, bradykinesia, rigidity, degradation of fine-motor skills, depressive tendencies, speech impairment and sleep abnormalities [1]. Despite being incurable for the Submission date: May 2, 2023 This work was supported by European Union’s Horizon 2020 research and innovation programme under grant agreement No 965231. Alexandros Papadopoulos ([email protected]) and Anastasios Delopoulos ([email protected]r) are with the ECE Department of Aristotle University of Thessaloniki. time being, the stage of the disease at first diagnosis can have a large impact on the disease progression and the patient’s quality of life, with early diagnosis linked to significant improvements in both of these outcomes [2]. This link, along with the fact that early signs of PD are often unnoticed or ignored by patients, has jumpstarted a line of research [3] into using machine learning (ML) on sensor data for continuously and objectively monitoring an individual for patterns or behavioural changes that might indicate PD. To this end, various sensor types and algorithms have been proposed, targetting different symptoms of the disease. For instance, tapping on virtual or physical keyboards has been used for quantifying fine-motor impairment [4]–[6], microphones for estimating the severity of speech degradation [7]–[9] and wearable accelerometers and gyroscopes for detecting tremor [10]–[12], gait imbalance [13]–[15] and eating difficulties caused by bradykinesia [16]. Building on the basic idea of associating data from a specific type of sensor to a specific motor symptom, some works even follow a more holistic approach and combine multiple sensors to detect PD itself and not just any one symptom [17]–[20]. Most approaches in the literature, collect data in a controlled lab-based or, at best, a scripted home-based environment. Few works actually operate in the free-living setup, despite being the most suitable for large scale adoption of any proposed technologies. The rather slow transition to the unobtrusive and in-the-wild setting is not without reason, as the latter poses significant difficulties to machine learning approaches, ranging from low signal-to-noise ratios to difficulties in obtaining precise ground-truth for training and evaluation. These difficulties are usually circumvented by using coarse, subjectlevel annotations that allow predictions to be made using multiple-instance learning [12], [21]–[23]. However, acquiring even a coarse annotation can be challenging when collecting data from hundreds of people, and each one must be subjected to a neurological evaluation. On the contrary, as we will see, collecting unlabeled data from hundreds of subjects is much easier and cheaper to accomplish. In this paper, we study the problem of predicting whether a person suffers from PD-induced tremor by analyzing hand acceleration signals that have been captured unobtrusively throughout the daily user-smartphone interaction. To this end, a data collection app that “listens” to the phone’s IMU sensor whenever a phone call is placed was employed (developed within the context of [24]). Remote screening for PD tremor arXiv:2305.00249v1 [cs.LG] 29 Apr 2023 2 under unobtrusive and free-living conditions was first tackled in [21] using a relatively small dataset of 45 subjects that were all annotated for tremor at the subject level. Here we examine whether tremor detection performance can be improved via the incorporation of an additional large dataset of 454 subjects, collected under the same conditions, but lacking tremor annotation. This is a natural next step, as acquiring additional unlabelled data is very easy: we just need to distribute the data collection app to as many people as possible. On the contrary, acquiring labelled data requires the person to be neurologically evaluated, a process both time-consuming and expensive and thus not scalable to large numbers of participants. Building ML models using few labelled and much unlabelled data corresponds to a learning approach called semisupervised learning. Semi-supervised learning (SSL) has received much attention from the research community over the years, owing to the ease of obtaining vast amounts of data and to the inherent difficulty in labelling them. In recent years, semi-supervised approaches for deep learning have led to impressive results. Approaches like self-supervised learning [25] and consistency-based regularization [26] before it, have been successfully applied to many image classification problems, achieving performance comparable to their fully-supervised counterparts, while using only a handful of labels [27]–[29]. Currently, most SSL approaches work in the single-instance learning setting, where the goal is to predict the label yof a data sample x. However, a different setting with particular interest for remote disease screening is that of MultipleInstance Learning (MIL), where the goal is to predict a label y for a bag of samples (or instances){x1,x2, . . . }. Throughout the learning procedure, the labels of the instances in the bag are unknown and the only available annotation is a label that describes the entire bag. This situation is regularly encountered in practice. In our case, for example, PD tremor may manifest only for a fraction of time during everyday life, depending on the disease’s stage, the symptom’s intermittence or levodopa intake. Hence, to detect tremor from sensor measurements obtained in-the-wild, we must resort to MIL, as there is no easy way to obtain detailed ground-truth of the on-off periods. Here we are interested in the combination of semisupervised with multiple-instance learning. In particular, we are interested in whether we can use unlabelled bags to improve a multiple-instance classifier. Interestingly, this problem has received very little research attention. The early work of [30] uses unlabelled bags in a content-based image retrieval setting and proposes a way of transforming the MI problem to a single-instance graph-based label propagation [31] problem that has the MI constraints encoded in the graph structure. The subsequent work of [32] uses a similar graph-based approach and suggests a unification between the representation of the image on the instance level and on the bag level to transductively annotate images. A further modification of the graph optimization objective is proposed in [33]. An interesting approach is the one by [34] who propose a regularizationbased MI SSL approach for video annotation tasks, in which similar instances are encouraged to share similar labels, while an instance-level label propagation scheme that combines label propagation with MI is proposed in [35]. Finally, [36] explores a scenario where a MIL classifier is trained using similarity information between bags, rather than complete labels. Based on the above overview, one notices that almost all of these works are concerned with transductive SSL [37], where the goal is to assign labels to the given unlabelled data (i.e. propagate labels to the bag instances), rather than learning a general mapping from the data to the label domain. In addition, most related methods predate deep learning and are focused on more traditional machine learning models. In this paper we follow a different direction and propose an approach based on consistency-regularization that is able to leverage unlabelled bags in order to improve a deep multiple-instance learning classifier. Concretely, we make the following contributions: (i) We propose a method for leveraging unlabelled bags of instances in multiple-instance problems, by adapting a state-of-the-art semi-supervised algorithm to the multiple-instance setting. Furthermore, we introduce two additional modifications to the main algorithm that further boost performance. (ii) To demonstrate its validity in a controlled environment, we conduct thorough experiments in syntheticallygenerated datasets based on the MNIST and CIFAR-10 datasets, where we demonstrate systematic improvements in performance from the incorporation of unlabelled data, compared to the fully supervised baseline and two alternative state-of-the-art SSL algorithms. (iii) We introduce a new dataset of hand acceleration recordings captured unobtrusively from PD patients and Healthy controls during free-living conditions. It consists of a small cohort of 45 labelled (with tremor groundtruth) and a large cohort of 454 unlabelled (without tremor ground-truth) subjects. It serves as an extension of a previous dataset and is made publicly available here. (iv) By utilizing this unlabelled cohort, we demonstrate that our proposed approach can lead to large and systematic improvements in PD tremor detection performance (up to ∼9% increase in F1-score) compared to the alternative that makes use of just the small labelled cohort, thus demonstrating its utility in remote disease screening. The rest of this paper is organized as follows. In Section II we present the related literature for the SSL and MIL domains, focusing mostly on approaches that are related to ours. In Section III we introduce our method for MI SSL and its proposed variants. Section IV presents the initial, proof-ofconcept experimental results. Then section V introduces the in-the-wild tremor detection problem and demonstrates how our approach can be applied to improve performance in this problem. Finally, in VI we discuss the potential benefits and caveats of the presented approach. II. PRELIMINARIES A. Semi-Supervised Learning Semi-supervised learning (SSL) is a situation where in addition to a fully labelled set Dl={(xi, yi)}L i=1, we are also presented with a set of unlabelled data points Du={xi}L+U i=L+1 drawn i.i.d. from the same marginal distribution. The goal is to leverage Duin order to learn a more accurate classifier than PAPADOPOULOS et al.:LEVERAGING UNLABELLED DATA IN MULTIPLE-INSTANCE LEARNING PROBLEMS FOR IMPROVED DETECTION OF PARKINSONIAN TREMOR IN FREE-LIVING CONDITIONS 3 what would be possible using only Dl. In general, it is not evident how unlabelled data can help, as knowledge of the marginal does not directly contribute to the data likelihood for a given model. In fact, Ducan be helpful only if certain assumptions are true. The most common one is the smoothness assumption, which states that if two points x1,x2in a highdensity region are close, then so should be their predicted labels y1,y2. This suggests that the learnt classifier must be smooth in high-density regions. The well-known low-density separation assumption that requires the decision boundary to lie in a low-density region, is an alternative view of the smoothness assumption. Early SSL techniques for neural networks were designed to enforce the low-density separation assumption on the resulting classifier. They did so by penalizing a decision boundary in high-density regions, for example by encouraging the model output distribution to have low entropy [38]. More recent techniques employ a similar regularization scheme, in which the model is encouraged to be invariant across label-preserving transformations of the input data (e.g. a small amount of additive gaussian noise that does not change the label). This principle is called consistency regularization and is used in many state-of-the-art methods for semi-supervised image classification, like Pseudo-Ensembles [39], Temporal Ensembling [40] and Mean Teacher [41]. One approach to consistency regularization that is of particular interest in this work, is the Virtual Adversarial Training (VAT) method [42]. Instead of perturbing xrandomly, VAT calculates the perturbation that will cause the maximum change in the model’s output. To achieve this, at each training step, it solves the following optimization problem: rvadv = arg max r;||r||2= Dhp(y|x;ˆ θ), p(y|x+r;ˆ θ)i(1) where p(y|x;ˆ θ)is our model, Dis a distribution divergence metric and ˆ θdenotes the model parameters at the current step. The optimization problem of Eq. 1 can be approximated efficiently with just an additional forward-backward pass through the network. Having estimated rvadv, the model is then encouraged to be smooth along its direction by minimizing the Local Distributional Smoothing (LDS) loss at each data point: LDS =Dhp(y|x;ˆ θ), p(y|x+rvadv;ˆ θ)i(2) Empirical results suggest that encouraging consistency along the virtual adversarial direction, rvadv, results in significantly improved performance compared to consistency in random perturbations. In the following, we will present an approach for semi-supervised multiple-instance learning that is based on VAT. We elect this particular technique because its ultimate goal is conceptually intuitive and elegant and its background is theoretically sound. B. Multiple-Instance Learning In Multiple-Instance Learning (MIL) we are again presented with a set of samples and their labels D={(Xi, yi)}L i=1. The difference is that here each sample is itself a bag instances, i.e. Xi={x1 i,x2 i,...,xK i}with xj i∈RN, while yirefers to the entire bag Xiand not to any one instance xj i. The goal in this scenario is to learn a bag classifier. Since a bag is an unordered set of instances without dependencies between its members, our classifier should be permutation-invariant with respect to the ordering of the bag instances. Theoretical results [43] suggest that a bag function f(X)is permutation-invariant if and only if it can be decomposed in the form: f(X) = ρ(z),z=X x∈X φ(x)(3) where φis an embedding function RN7→ RM,z∈RMis the embedding of Xand ρa suitable transformation RM7→ Y (with Ywe denote the label domain). For models based on neural networks, the transformation φis usually a high-capacity CNN that is used either for feature extraction or for direct instance classification. The transformation ρis then either a classification head that takes us from the embedding space to the class space, or simply the identity. A rather interesting modification to the above stems from incorporating an attention mechanism on the sum of Equation 3. This approach [44] defines the bag embedding as a non-linear combination (with learnable parameters V,w) of the instance features: z= K X k=1 αkφ(xk), αk=ewTtanh(Vφ(xk)T) PK i=1 ewTtanh(Vφ(xi)T)(4) The attention parameters Vand wcan be easily modelled as neural networks, thus allowing the whole model to be learnable end-to-end. In addition, attention scores provide an elegant way of identifying key instances within a bag. Attention-based MIL has been successfully applied to many problems [22], [45], [46]. Owing to its attractive properties and performance, we will use it as our core MIL model, which we will enhance in the next section with a semi-supervised component. III. SEMI-SUPERVISED MULTIPLE-INSTANCE LEARNING In this section, we present our approach for utilizing unlabelled bags of instances in order to improve a multipleinstance classifier. As VAT provides a principled and elegant way of using unlabelled data, we elect to use it for semisupervised MIL, over other SSL approaches (e.g. Mean Teachers) that could be directly applied to the same problem. First, we extend VAT to the multiple-instance scenario. To that end, we introduce the concept of bag perturbation which is a set R= (r1,r2,...,rK)that when added elementwise to a given bag X, slightly perturbs it. The Multiple-Instance Local Distributional Smoothing (MI-LDS) loss can now be defined as: MI-LDS(X, ˆ θ) = Dhp(y|X;ˆ θ), p(y|X+Rvadv;ˆ θ)i(5) where X= (x1,x2, . . . xK) Rvadv = (r1,r2,...,rK) = arg max R;||rk||2<= D[p(y|X;ˆ θ), p(y|X+R;ˆ θ)] X+Rvadv = (x1+r1,x2+r2,...,xK+rK) 4 and Dis a distribution divergence like KL divergence. As MI-LDS does not depend on the bag label, it can be added as an unsupervised regularization term in the loss function. Assuming a set of labelled bags Dl={(Xi l, yi l)}|L i=1 and a set of unlabelled bags Dul ={Xj ul}|U j=1, the loss function is L=E X,y∼Dlh−log p(y|X;ˆ θ)i+E X∼Dul hMI-LDS(X, ˆ θ)i (6) where the first term is the standard cross-entropy loss computed over labelled bags and the second term is the MI-LDS loss computed over the unlabelled bags. We will now show how to compute Rvadv based on the original approximation method of the virtual perturbation direction in the single-instance scenario [42]. Without loss of generality, we will treat Ras a flattened vector of the form [r1 1, . . . , r1 N, r2 1,...r2 N, . . . , rK 1,...rK N]. For brevity, we denote Dhp(y|X;ˆ θ), p(y|X+Rvadv;ˆ θ)iby DR, X, ˆ θ. We also assume that p(y|X;ˆ θ)is differentiable twice with respect to ˆ θ and Xalmost everywhere. Since DR, X, ˆ θis minimized at R0= (0,0,...,0)and given the differentiability assumption, we have that ∇RD(R, X;ˆ θ)|R=R0=0. Hence, its secondorder Taylor approximation at R=R0is given by DR, X, ˆ θ≈1 2RTH(X;ˆ θ)R(7) where H(X;ˆ θ) := ∇∇RDR, X, ˆ θ|R=R0is the Hessian matrix. Under this approximation, computing Rvadv reduces to computing the first dominant eigenvector of the Hessian. This can be achieved by means of the power-iteration approach: we simply start out with a randomly sampled perturbation Vand compute the following product d←HV (8) where the overline operator ddenotes normalization of each element of d(which here is a bag entity) to unit length. Repeating this calculation iteratively makes dconverge to the first dominant eigenvector of H. Interestingly enough, the HV product can be approximated via a finite difference method, thus eliminating the need for directly computing H: HV ≈∇RD(R, X, ˆ θ)|R=ξV − ∇RD(R, X, ˆ θ)|R=R0 ξ(9) =∇RD(R, X, ˆ θ)|R=ξV ξ(10) where ξis a small constant. Putting it all together, we can approximate Rvadv with the repeated calculation of V← ∇RDR, X, ˆ θ|R=ξ·V(11) where initially Vis a bag of randomly sampled unit vectors. In each iteration of Equation 11 we first compute the gradient of Don R=ξV and then normalize each element to unit length. Based on the above, the virtual adversarial bag perturbation Rvadv can be approximated using the one-time power iteration approach of Algorithm 1. Algorithm 1: Approximation of Rvadv with a one-time power iteration approach Input : Bag X= (x1,x2, . . . xK), model p(y|X;θ) Output: Virtual adversarial bag perturbation Rvadv 1) Generate a random bag V= (v1,v2, . . . vK)with vk∼N(0, I) 2) Compute p(y|X;ˆ θ)and p(y|X+ξV ;ˆ θ) 3) Compute the (Kullback-Leibler) divergence L=DKL p(y|X;ˆ θ)kp(y|X+R;ˆ θ) 4) Calculate the gradient of Lon R=ξV : G= (g1,g2,...,gK)← ∇RL|R=ξ·V 5) Normalize elementwise to the desired magnitude. Rvadv ←g1 ||g1||2 ,  g2 ||g2||2 , . . . ,  gK ||gK||2 This procedure results in a bag perturbation that is dense, meaning that all instances in the input bag Xare going to be perturbed: Variant 1 - Regular (dense) perturbation Rvadv = ( r1,r2,...,rK | {z } K non-zero instances ) An interesting variation of this approach would be to make Rvadv sparse, i.e. restrict it so that it has a single non-zero instance. The hope in doing so is that by perturbing one bag instance instead of all, we will arrive at more meaningful bag perturbations overall. This is due to the fact that in MIL, changing the (unobserved) label of a single instance can change the label of the bag itself, as a positive bag can contain as little as a single positive instance (in the case of the standard MI assumption [47]). Therefore, in such cases it may prove more beneficial to perturb the few positive instances of a positive bag, rather than all of its instances indiscriminately. In practice, however, we do not know which are the positive instances in a bag in order to perturb them in a targeted manner. Thus, to mitigate this obstacle we opt for a stochastic approach, in which we select the index jof the instance to be perturbed at random. We propose two different strategies for choosing j. The first approach is to simply select juniformly at random over the Kpossible instances. This approach leads to the following perturbation variant: Variant 2 - Sparse perturbation with uniform probability Rvadv = (0, . . . 0,rj,0, . . . 0 | {z } single non-zero instance ) where p(j=k)=1/K, 1≤k≤K Let us now assume that p(y|X;ˆ θ)is the attention-based MIL model defined in Equations 3 - 4, that is: py|X;ˆ θ=ρ X k αkφ(xk)!(12) PAPADOPOULOS et al.:LEVERAGING UNLABELLED DATA IN MULTIPLE-INSTANCE LEARNING PROBLEMS FOR IMPROVED DETECTION OF PARKINSONIAN TREMOR IN FREE-LIVING CONDITIONS 5 Feature extractor MIL Attention KxM Instance embeddings 1xM Classifier KxN Bag of instances Bag embedding (a) The attention-based deep MIL model. Bag perturbation Regularizer Adversarial perturbation (b) The bag perturbation idea in a nutshell. Fig. 1: High-level overview of the main components involved in the proposed semi-supervised MIL methodology. where φ, ρ, αkare learnable. Given this model, a third variant for the perturbation form is to sample jfrom a multinomial distribution whose parameters are given by the attention weights αk, as {αk}defines a multinomial distribution over Koutcomes since 0≤αk≤1and Pkαk= 1. This can be written concisely as: Variant 3 - Sparse perturbation based on attention Rvadv = (0, . . . 0,rj,0, . . . 0 | {z } single non-zero instance ) where p(j=k) = αk(per Eq. 4),1≤k≤K In this way, the non-zero index jis sampled from a different distribution for each bag, with instances that receive high attention (which hopefully correspond to key instances) more likely to be selected in each training step. In the following, we will explicitly use the attention-based model as our underlying p(y|X;ˆ θ), because, apart from its attractive properties and performance, it allows us to seamlessly accommodate variants 2 and 3 in Algorithm 1 by means of a simple masking system. A high-level overview of the components involved in the proposed methodology is presented in Fig. 1. IV. PROOF-OF-CONCEPT EXPERIMENTS Prior to addressing tremor detection in-the-wild, we wish to establish the validity of our approach in a more controlled environment. To that end, in this section we conduct some initial proof-of-concept experiments on two semi-supervised multiple-instance problems, that are synthetically generated from two standard image classification datasets: •MNIST 1, a dataset of 70,000 28x28 grayscale images 1http://yann.lecun.com/exdb/mnist/ of handwritten digits (0 to 9), split in a training set of 60,000 and a test set 10,000 images. •CIFAR-10 2, a dataset containing 60,000 32x32 color images of the following mutually exclusive classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. Its training/test split is 50,000/10,000. To create MIL problems from these datasets we will define a bag as a set of random images. For MNIST, a bag will be positive if it contains at least one image of the digit ”9” (as per [44]), while for CIFAR-10, a positive bag will have to contain at least one image from the ”Truck” class. The length Kof each bag is randomly sampled from the Gaussian distribution N(Kmean , Kstd ). The number of positive samples in a positive bag is sampled from the uniform distribution U(1, K), while class imbalance is controlled by a parameter p1. For our experiments, we create a training set of Llabelled and Uunlabelled bags from the official training set of each dataset. The instances in each bag are sampled without replacement, so that bags do not contain common instances. For testing, we sample a set of 1.000 bags from the official test set using the same parameters. For efficiency, we opt for relatively small bag sizes, thus setting Kmean =10 and Kstd =2 in the data generation process (larger bag sizes arise naturally in the context of our tremor detection problem of Section V). In addition, positive class imbalance was set to 1 positive for every 10 negative (p1=0.1) in order to simulate the case of highly imbalanced classes that is encountered in real-world datasets (see Section V). For MNIST, we used a LeNet-5 model for the embedding function φ(in accordance with [44]) with an embedding of size M=800. The instance embeddings were then merged via an attention mechanism with L=128 and the resulting bag 2https://www.cs.toronto.edu/˜kriz/cifar.html 6 TABLE I: Average ROC-AUC scores across 10 trials of the MNIST experiment. Positive bags are those with at least one ”9” digit. Labelled Unlabelled Method ROC-AUC 50 0Baseline 0.702 ±0.057 200 Temporal ensembling [40] 0.768 ±0.019 Mean teacher [41] 0.833 ±0.025 Dense MI-VAT 0.859 ±0.043 Sparse-Uniform MI-VAT 0.927 ±0.020 Sparse-Attention MI-VAT 0.943 ±0.025 400 Temporal ensembling [40] 0.734 ±0.048 Mean teacher [41] 0.903 ±0.012 Dense MI-VAT 0.881 ±0.040 Sparse-Uniform MI-VAT 0.918 ±0.029 Sparse-Attention MI-VAT 0.935 ±0.033 100 0Baseline 0.811 ±0.052 200 Temporal ensembling [40] 0.904 ±0.026 Mean teacher [41] 0.926 ±0.010 Dense MI-VAT 0.973 ±0.026 Sparse-Uniform MI-VAT 0.986 ±0.008 Sparse-Attention MI-VAT 0.984 ±0.009 400 Temporal ensembling [40] 0.872 ±0.020 Mean teacher [41] 0.947 ±0.019 Dense MI-VAT 0.988 ±0.009 Sparse-Uniform MI-VAT 0.980 ±0.012 Sparse-Attention MI-VAT 0.982 ±0.011 TABLE II: Average AUC scores across 5 trials of the CIFAR10 experiment. Positive bags are those with at least one image of the ”Truck” class. Labelled Unlabelled Method ROC-AUC 200 0Baseline 0.732 ±0.067 400 Temporal ensembling [40] 0.762 ±0.049 Mean teacher [41] 0.795 ±0.033 Dense MI-VAT 0.858 ±0.008 Sparse-Uniform MI-VAT 0.833 ±0.031 Sparse-Attention MI-VAT 0.869 ±0.021 800 Temporal ensembling [40] 0.743 ±0.037 Mean teacher [41] 0.777 ±0.026 Dense MI-VAT 0.865 ±0.021 Sparse-Uniform MI-VAT 0.874 ±0.027 Sparse-Attention MI-VAT 0.902 ±0.019 400 0Baseline 0.860 ±0.029 400 Temporal ensembling [40] 0.769 ±0.054 Mean teacher [41] 0.851 ±0.020 Dense MI-VAT 0.919 ±0.008 Sparse-Uniform MI-VAT 0.920 ±0.016 Sparse-Attention MI-VAT 0.917 ±0.014 800 Temporal ensembling [40] 0.759 ±0.030 Mean teacher [41] 0.856 ±0.017 Dense MI-VAT 0.928 ±0.012 Sparse-Uniform MI-VAT 0.919 ±0.011 Sparse-Attention MI-VAT 0.928 ±0.006 embedding is transformed to a class score via a single linear layer (transformation ρ). For CIFAR-10, we used the ConvSmall architecture from [42] with M=192 for φ,L=128 and a single linear layer for ρ. To gain some insight about the behaviour of our MI-VAT variants under various conditions, we ran the experiment for multiple values of labelled bags Land unlabelled bags Uand compared them against i) a baseline model of the same architecture and training details that makes no use of unlabelled data, and ii) Mean Teacher and Temporal Ensembling, two alternative SoA SSL that can be directly applied on the same problem scenario by incorporating their respectively proposed regularization terms in the overall cost function, as these terms depend only on the model’s past predictions. Both approaches were trained on the exact same data and tuned using the same budget for hyperparameter tuning. For each Land Uvalue, we train the model 10 times and compute the average Area Under Receiver Operating Curve (ROC-AUC) across all trials. All models were trained for 100 epochs using the Adam optimizer with a base learning rate of 0.001, except for Mean Teacher that was trained using SGD as per [41]. The results for both experiments are presented in Tables I and II. Based on these results, we see that MI-VAT can lead to large performance improvements, beating both the baseline and the alternative SSL approaches that were considered. More specifically, in the MNIST experiment we see improvements of up to 24 AUC points over the baseline (Sparse-Attention -L=50 U=200), while in the CIFAR-10 experiment we see improvements of up to 17 points (Sparse-Attention - L=200 U=800). Regarding the 3 MI-VAT variants, we see that SparseUniform and Sparse-Attention generally result in better performance than Dense MI-VAT, with Sparse-Attention often resulting in the best performance overall. In addition, we also notice that the performance improvement tends to be larger for smaller numbers of labelled bags. V. IMPROVED DETECTION OF PD TREMOR IN-THE-WILD Having established the validity of our approach in a controlled setting, we now turn our attention to the actual problem that this research effort aspires to tackle. First introduced in [12] and [21], the problem consists of analyzing hand acceleration signals collected during the daily interaction of a person with their smartphone, in an attempt to infer if the person exhibits hand tremor caused by PD. To this end, a dataset of many acceleration recordings from 45 subjects was collected via a dedicated smartphone app that was installed on the participants’ personal smartphones. After installation, the app recorded inertial data whenever a phone call was placed. Most importantly, data collection was carried out in-the-wild, that is, it ran during unscripted and free-living conditions, without any interaction or requirement from the user, following an ‘install and forget it’ principle. All 45 participants, both diagnosed PD patients and healthy controls, were recruited in the context of i-PROGNOSIS project [24] and were examined by medical experts for hand tremor and other PD symptoms. In these earlier works, the goal was to build a tremor detection model using only data from the 45 ”labelled” subjects. As tremor is an intermittent symptom by nature, its appearance in the bulk of the acceleration signals contributed by a PD patient can be a relatively sparse event. Furthermore, as the data were collected completely unobtrusively and in-the-wild, there was no way of knowing when the tremorous episodes took place. Thus, the problem was naturally framed as a MIL classification problem with two classes: tremor vs no tremor at the participant level. Under this view, a participant PAPADOPOULOS et al.:LEVERAGING UNLABELLED DATA IN MULTIPLE-INSTANCE LEARNING PROBLEMS FOR IMPROVED DETECTION OF PARKINSONIAN TREMOR IN FREE-LIVING CONDITIONS 7 TABLE III: Basic demographic characteristics of the labelled and unlabelled cohorts used in the tremor detection experiment. Provided values are the mean of the population with the standard deviation in parentheses. PD status for the unlabelled data was self-reported by the participants themselves and not officially provided by neurologists. Tremor positive refers to whether a subject exhibits hand tremor or not (not all PD patients do exhibit tremor). Labelled data (introduced in [21]) Unlabelled data Healthy PD Total Healthy (self-reported) PD (self-reported) Total No. of participants 14 31 45 380 74 454 Age 55.4 (11.7) 62.1 (7.3) 60.0 (9.4) 54.1 (10.3) 60.3 (8.3) 55.1 (10.2) Years after PD diagnosis - 6.3 - - NA - UPDRS 16 0.07 (0.25) 1.09 (0.89) - NA NA - UPDRS 20 0 (0) 1.19 (1.25) - NA NA - UPDRS 21 0 (0) 0.96 (1.51) - NA NA - Sum of UPDRS-III 2.28 (3.47) 19.7 (11.5) - NA NA - Contributed acceleration segments 441 (512) 576 (537) 534 (533) 343 (347) 278 (319) 333 (344) Tremor positive 0 16 16 NA NA NA Fig. 2: Schematic overview of the data collection and pre-processing operations that take place prior to the actual tremor detector training. The data of a subject, collected unobtrusively during free-living conditions, are transformed via a 3 stage pipeline (pre-processing, segmentation and post-processing stages) to a bag of acceleration segments. was represented as a bag of acceleration signal segments, accompanied by a single label that indicated whether they exhibited tremor in general or not. An attention-based MIL approach was then used to train a binary classification model that resulted in very good discrimination of tremorous vs non-tremorous participants. The demographics of this cohort are provided in the left half of Table III. The actual tremor label was created through manual inspection of the contributed signals of a subject, in conjunction with their neurological evaluation scores. This was necessary because using directly the clinical scores as ground-truth can lead to label noise in a variety of situations [21], caused by the unobtrusive data collection, the intermittent/unilateral nature of tremor and the sparsity of the neurological evaluation combined. To form the bag of segments encoding for a subject, a preprocessing pipeline was applied, during which an accelerometer signal (or session) was discarded if some conditions were met (short duration, low sampling frequency fs, extreme values, etc). If not, it was resampled to a common fsof 100Hz and 5s were trimmed from both signal ends to remove picking/hanging up gestures. Finally, a high pass filter was applied to remove the gravitational component of the acceleration signal. The transformed signal was then segmented to non-overlapping 5s segments. The segments from the various signals of a subject were then ranked according to their energy in the PD tremor band (3-7Hz) and the top Ktsegments were kept to form the bag of segments for a subject. The overall process is depicted schematically in Fig. 2. Apart from directly recruiting participants, the data collection app was also made available in the Android app store for anyone to download and use. People that contributed data through that channel were not examined by doctors, so there was not any tremor ground-truth available for them. Ultimately, this parallel data collection led to a second cohort of 454 people who contributed acceleration data but without any labels, apart from a self-reported PD status. As a result, this larger cohort was not employed in [21] due to the lack of tremor ground-truth. The demographics of this cohort are provided in the right half of Table III. In this paper, we wish to examine if the unlabelled data from the large cohort can be used in a semi-supervised manner to improve the performance of a MIL tremor detector. To this end, we follow the approach of [21] and encode each subject as a bag of acceleration segments using the same preprocessing pipeline that was described above but we keep the top K=100 segments (instead of K=1500 that was used in [21]) for computational efficiency purposes, as well as to make the problem harder. After encoding each subject as a bag, we extract features from the raw tri-axial acceleration segments via a function φthat is a fully-convolutional 1D CNN with 4 layers, followed by average pooling and a fully-connected layer, that leads to an embedding of M=64 dimensions. We also use an attention dimension of L=128 and a fullyconnected network with 3 layers for the transformation ρ. The model architecture is concisely given in Table IV. We chose this particular architecture based on the literature and our 8 TABLE IV: Instance embedding transformation φand classification head ρ.kdenotes the kernel size, fthe number of filters, sthe stride and Mthe embedding dimension. φ ρ Input x∈ R3×500 acceleration segment z∈ R64 Layer 1 Conv1D k= 4, f = 32, s = 2 Dense M→32 Leaky-ReLU (α= 0.2) Leaky-ReLU (α= 0.2) Dropout p= 0.2 Layer 2 Conv1D k= 4, f = 64, s = 2 Dense 32 →10 Leaky-ReLU (α= 0.2) Leaky-ReLU (α= 0.2) Dropout p= 0.2 Layer 3 Conv1D k= 4, f = 128, s = 2 Dense 10 →2 Leaky-ReLU (α= 0.2) 2-way softmax Dropout p= 0.2 Layer 4 Average 1D Pooling Dense 128 →64 Output hk∈ R64 p(y|X) TABLE V: Semi-supervised classification results for in-thewild PD tremor detection. We report the average performance metrics across all LOSO iterations and random trials. Method Precision Specificity Sensitivity F1-score Baseline [21] 0.854 0.917 0.875 0.864 Dense 0.938 0.966 0.938 0.938 Sparse-Uniform 0.904 0.945 0.938 0.920 Sparse-Attention 0.950 0.972 0.950 0.950 previous experience in the field. Thus, starting from the model of [21], which exhibited very high performance on the tremor detection task, we introduced two modifications: replacing max-pooling with strided convolutions and flatten/dense layers with global average pooling. This was done in order to reduce training time without sacrificing model capacity, as MI-VAT introduces a non-trivial computational overhead. For our classification experiment we follow a Leave One Subject Out (LOSO) evaluation, in which we use the data from all labelled participants except one to train a model, which we then evaluate on the left-out person. The process is repeated until all participants have been left out and the results from each iteration are averaged. For the baseline approach, we train the model in a fully-supervised manner using only the labelled data, while for the SSL approach we use both labelled and unlabelled data as per Eq. 6. We do not compare against the alternative SSL approaches here, as they did not offer significant improvement over the baseline in our, more controlled, proof-of-concept experiments. At each LOSO split, we repeat model training 5 times to reduce biases caused by the random initialization in network weights. All models were trained for 100 epochs using the Adam optimizer with a base learning rate of 0.0003. We compute the average Precision, Sensitivity, Specificity and F1-score for a decision threshold of 0.5(as per [21]) across all splits and random trials. The results of this experiment are reported in Table V. We can see that all MI-VAT variants offer significant improvements over the baseline, with the Sparse-Attention variant, in particular, leading to the best performance overall with an F1-score almost 9% higher than the baseline model. 0 100 200 300 454 80 85 90 95 100 86.4 87.2 91.0 90.5 95.0 Subjects included in the unlabelled set F1-score (%) Fig. 3: Tremor detection performance under different unlabelled set sizes. Each y-value represents the average F1score computed over 10 randomly-sampled subsets of the full unlabelled set, that are stratified with respect to the selfreported PD status of the subjects. The standard deviation across trials is also presented via error bars. Finally, to study the effect of incorporating unlabelled data in our tremor detection pipeline, we conduct an additional experiment, where we limit the size of the unlabelled set to just 100, 200 and 300 subjects. For each of these values, we randomly sample 10 subsets from the full unlabelled set that are stratified with respect to the self-reported PD status, to retain approximately the composition of PD patients and Healthy subjects. Then, we train the Sparse-Attention variant using the resulting subset and compute the average F1-score across all 10 subsets for each unlabelled set size that we examine. The results of this experiment are presented graphically in Fig. 3. Based on these results, it becomes evident that there is a clear connection of increased size of unlabelled data with improved performance, although this relation is not of a strictly increasing nature, as exemplified by the small performance decrease between 200 and 300 subjects. VI. DISCUSSION We began evaluating MI-VAT through controlled experiments on synthetic datasets. There, we established that it can indeed leverage unlabelled data to improve classification performance. An additional finding from those experiments, that agrees with our intuition, is that performance tends to improve when we increase the number of unlabelled bags (Tables I, II). This was also the case in the real-world tremor problem we examined later (Fig. 3), although it appears that the relation between performance and unlabelled set size is not strictly increasing. Stepping on these results, we then turned our attention to the real problem at hand: using real-world unlabelled data to improve PD tremor detection in-the-wild. To that end, we introduced a novel, unlabelled dataset of acceleration PAPADOPOULOS et al.:LEVERAGING UNLABELLED DATA IN MULTIPLE-INSTANCE LEARNING PROBLEMS FOR IMPROVED DETECTION OF PARKINSONIAN TREMOR IN FREE-LIVING CONDITIONS 9 recordings from 454 subjects that was used in conjunction with an already available dataset of 45 labelled subjects. Through a repeated LOSO experiment, we showed that our MI-VAT approach leads to significant improvements over the fullysupervised baseline of [21]. More specifically, we observed an increase in F1-score of ∼7.5% for Dense, ∼6% for SparseUniform and ∼9% for Sparse-Attention variants, a result that further reinforces our belief that the latter variant is the most beneficial. The importance of successfully leveraging unlabelled data of this type is not limited to the field of PD. In fact, it paves the way for incorporating unlabelled data in remote, data-driven, screening pipelines for other diseases, like Alzheimer’s or depression, which also benefit from a MIL treatment, as their symptoms can too be intermittent. Having said that, our approach, at its current form, suffers from one important limitation: it cannot be directly applied to any data type, due to the use of additive noise for instance perturbation. While an additive perturbation may be perfectly fine for data like images or, more generally, signals that live in vector spaces like RN, it may be unsuitable for others, like the typing data of [22] where the bag instances were histograms of button hold and release intervals. Using an additive perturbation here, would be problematic, because the perturbed vector would cease to be a histogram (its values would no longer sum to 1). Therefore, in such cases, modifying the perturbation operation to ensure that the perturbed instance does not fall outside the data manifold is necessary. Apart from being an interesting problem from an academic perspective, the extension of our semi-supervised approach to additional data types is crucial for its introduction into clinical practice of remote PD screening. This is because patients typically exhibit only a subset of the total PD symptomatology, thus making it difficult to accurately diagnose the condition by looking for signs of just a single symptom. Therefore, in order for a remote screening tool to be successful, it must evaluate an individual for as many symptoms of PD as possible. This can be achieved by analyzing data from a variety of sensors that have been shown to contain diagnostic information for specific PD symptoms. Such data sources for example, include typing patterns collected via smartphone virtual keyboards that have been linked with fine motor degradation, or speech recordings that may reveal signs of voice degradation caused by PD. Analysis of data from these sources could also benefit from a MIL treatment, as the corresponding PD symptom may not be always observable especially in the early stages of the disease. In addition, just as with the IMU-centric approach that we described here, collecting just unlabelled data will be much easier than acquiring ground-truth by neurologists. The proposed approach is further limited by the inherent constraints of semi-supervised learning. For a semi-supervised method to be effective, it is crucial that the labelled and unlabelled datasets are created in a similar manner, that is, they are generated from the same underlying data distribution. Unfortunately, this may not always be the case in practical scenarios. For example, it might as well be the case that the labelled dataset is collected from one country, while the unlabelled dataset comes from another country with distinct cultural practices and habits, thus leading to a shift in the data distribution between the two datasets. This potential complication must be thoroughly addressed during the design phase to ensure a successful real-world deployment of a monitoring system with a semi-supervised component, like the one proposed here. VII. CONCLUSIONS Motivated by the natural coexistence of labelled and unlabelled data in remote PD screening, we introduced a novel approach that unifies semi-supervised and multiple-instance learning efficiently. We first established its validity through controlled experiments on synthetic datasets. We then went on to significantly improve PD tremor detection in-the-wild, by leveraging IMU data from a large cohort of unlabelled subjects. To the best of our knowledge, this is the first work that uses unlabelled data to improve a MIL classifier in such a way. The consistently improved performance across all experiments, but, more importantly, its successful application on the real-world problem of remote and unobtrusive tremor detection, highlights the utility of the proposed methodology. APPENDIX One issue that remains unclear is the underlying mechanism of MI-VAT. In regular SSL, there is always some assumption about the data distribution that the training procedure tries to enforce on the model’s decision boundary. On a first level, it could be argued that the observed benefit in our case, stems from the consistency regularization effect of the MI-LDS loss term. However, this does not directly translate to constraining the shape of the decision boundary, because in MIL the decision boundary is not defined for individual instances but for sets of instances. Hence, thinking in terms of decision boundary is not intuitive. (a) Supervised - AUC=0.55 (b) Sparse-Attention - AUC=0.94 Fig. 4: Visualization of the learned bag embeddings zfor a toy MIL problem based on the two-moons dataset. To gain some intuition about this, we conduct a final exploratory experiment based on the two-circles dataset, a toy 2d dataset with a spherical decision boundary. We sample 50 labelled, 400 unlabelled bags and 1000 test bags, without replacement from a two circles dataset instantiation of 50,000 points. We then visualize (Fig. 4) the learned bag embeddings of a fully-supervised and a Sparse-Attention model using a φ with 3 fully-connected layers of 50, 30 and 2 hidden units. We can see that the MI-LDS objective has a significant effect on z, leading to embeddings that live in two distinct clusters. On the contrary, using just the cross-entropy loss leads to z with large inter-class intersection. Thus, it could be said that MI-LDS acts as a regularizer that encourages the inter-class bag embeddings to become better separated.