Mobile app recommendations using deep learning and big data

i Mobile App Recommendations Using Deep Learning and Big Data Author: Luís António Galego Pinto Advisor: Prof. Roberto Henriques Dissertation presented as partial requirement for obtaining the Master’s degree in Statistics and Information Management. i Title: Mobile App Recommendations Using Deep Learning and Big Data Luís António Galego Pinto MEGI 2018 MGI i ii NOVA Information Management School Instituto Superior de Estatística e Gestão de Informação Universidade Nova de Lisboa MOBILE APP RECOMMENDATIONS USING DEEP LEARNING AND BIG DATA by Luís Pinto Dissertation presented as partial requirement for obtaining the Master’s degree in Statistics and Information Management, with a specialization in Marketing Research and CRM. Advisor: Prof. Roberto Henriques iii ABSTRACT Recommender systems were first introduced to solve information overload problems in enterprises. Over the last decades, recommender systems have found applications in several major websites related to e-commerce, music and video streaming, travel and movie sites, social media and mobile app stores. Several methods have been proposed over the years to build recommender systems. The most popular approaches are based on collaborative filtering techniques, which leverage the similarities between consumer tastes. But the current state of the art in recommender systems is deeplearning methods, which can leverage not only item consumption data but also content, context, and user attributes. Mobile app stores generate data with Big Data properties from app consumption data, behavioral, geographic, demographic, social network and user-generated content data, which includes reviews, comments and search queries. In this dissertation, we propose a deep-learning architecture for recommender systems in mobile app stores that leverage most of these data sources. We analyze three issues related to the impact of the data sources, the impact of embedding layer pretraining and the efficiency of using Kernel methods to improve app scoring at a Big Data scale. An experiment is conducted on a Portuguese Android app store. Results suggest that models can be improved by combining structured and unstructured data. The results also suggest that embedding layer pretraining is essential to obtain good results. Some evidence is provided showing that Kernel-based methods might not be efficient when deployed in Big Data contexts. KEYWORDS Spark; Tensorflow; Social Networks; Machine Learning. iv INDEX 1. Introduction .................................................................................................................. 1 2. Literature Review ......................................................................................................... 4 2.1. Recommender Systems ......................................................................................... 4 2.1.1. Modeling Approaches .................................................................................... 4 2.1.2. Model Evaluation ........................................................................................... 9 2.1.3. Feature Extraction ........................................................................................ 16 2.2. Deep Learning ...................................................................................................... 18 2.3. Big Data ................................................................................................................ 21 3. Methods...................................................................................................................... 22 3.1. Proposed Model .................................................................................................. 23 3.2. Experimental Setup ............................................................................................. 24 3.3. Model Evaluation ................................................................................................. 26 3.4. Dataset Details..................................................................................................... 28 3.4.1. Raw Data....................................................................................................... 28 3.4.2. Feature Engineering ..................................................................................... 28 4. Results......................................................................................................................... 30 4.1. Model Training and Validation ............................................................................ 30 4.2. Model Testing ...................................................................................................... 31 5. Conclusion .................................................................................................................. 33 5.1. Discussion ............................................................................................................ 33 5.2. Implications ......................................................................................................... 35 5.3. Limitations ........................................................................................................... 36 5.4. Directions for Future Research ............................................................................ 37 References ....................................................................................................................... 39 Appendix.......................................................................................................................... 53 Appendix A – Mathematical Derivation of App Download Value .............................. 53 v LIST OF FIGURES Figure 1 – Perceptron ............................................................................................................... 19 Figure 2 – Shallow Network (Multilayer Perceptron) .............................................................. 19 Figure 3 – Deep Network ......................................................................................................... 19 Figure 4 – Recommendations displayed in the Aptoide mobile app home. ............................ 22 Figure 5 – Deep-Learning Architecture Overview .................................................................... 23 Figure 6 – Architecture of Model 1 and 3 ................................................................................ 25 Figure 7 – Architecture of Model 2 .......................................................................................... 25 Figure 8 – Architecture of Model 4 .......................................................................................... 25 vi LIST OF TABLES Table 1 – Features .................................................................................................................... 28 Table 2 – Computational Statistics of Model Training, Validation and Testing. ...................... 31 Table 3 – Computational Statistics of Model Deployment ...................................................... 31 Table 4 – Experimental Results of Model Testing (in detail) ................................................... 32 vii LIST OF ABBREVIATIONS AND ACRONYMS 3Vs Refers to the three properties of Big Data: Volume, Velocity and Variety. ALS Alternating Least Squares. ANN Artificial Neural Network. ARPU Average Revenue per User. ASO App Store Optimization. AUC Area Under the Receiver Operating Curve. BB Beta-Binomial. BG Beta-Geometric. CBR Case-Based Reasoning. CENE Content-Enhanced Network Representation Learning. CF Collaborative Filtering. CLV Customer Lifetime Value. CRM Customer Relationship Management. CTR Click Through Rate. DMD Dirichlet Multinomial Distribution. DNC Differentiable Neural Computer. DNN Deep Neural Network. E-Commerce Electronic Commerce. FC Fully Connected. GP-GPU General Purpose Graphics Processing Unit. HDFS Hadoop Distributed File System. IS Information Systems IT Information Technology KLR Kernel Logistic Regression. KNN K-Nearest Neighbors. KS Kolmogorov-Smirnov. 6 (Stewart, 1993). Suppose we have a real matrix 𝐴 with 𝑖 rows and 𝑗 columns, that represent the relationship between users and items, users and users, or users and their attributes. Our goal is to obtain a decomposition of 𝐴, such that: 𝐴=𝑈Σ𝑉𝑇 𝑈=(𝑢1,𝑢2,…,𝑢𝑖) 𝑉=(𝑣1,𝑣2,…,𝑣𝑗) Where Σ=𝑑𝑖𝑎𝑔(𝜎1,𝜎2,…𝜎𝑛) has nonnegative diagonal elements arranged in descending order of magnitude (the eigenvalues) and 𝑈 and 𝑉 are two matrices. The matrix 𝑈 contains the left eigenvectors, while 𝑉 contains the right eigenvectors (Gass & Rapcsák, 2004). If 𝐴 is a user by user matrix, 𝑈=𝑉. In cases where we have a user by attribute or user by item matrix, 𝑈 can be interpreted as the eigenvectors of the rows, while 𝑉 are the eigenvectors of the columns (which can be either items or user attributes). For the final analysis, we should select the relevant matrix of eigenvectors to use as the extracted eigenvectors (which can be either the left or the right depending on the definition of 𝐴 and the specific problem). The extracted eigenvectors can be interpreted as latent factors that explain the variance or inertia (Greenacre, 1988) of either the rows or columns of 𝐴 . The variance or inertia of each factor is nothing more than the eigenvalue (an element of Σ) associated with that eigenvector. These extracted dimensions have metric properties and can then be used to estimate an Euclidean distance between objects before applying KNN. The alternative is to use these as input to another model to score items for a specific user. Several applications of SVD to recommender systems are known (BarragánsMartínez et al., 2010; Brand, 2003; Paterek, 2007; Sarwar, Karypis, Konstan, & Riedl, 2000). Other similar approaches based on latent factors include Matrix Factorization using Alternating Least Squares (ALS) and Stochastic Gradient Descent (SGD) learning algorithms (Koren, Bell, & Volinsky, 2009). These approaches are like SVD but tend to generalize better to new cases. This is so because the learning is done using a numerical optimization algorithm which is not only able to better deal with missing values but can also include a regularization term. In these approaches we may have a model as such: 𝑟𝑖𝑗=𝑣𝑗𝑇𝑢𝑖+𝜇𝑖+𝑏𝑖+𝑏𝑗 Where 𝑟𝑖𝑗 is the predicted rating of item 𝑖 for user 𝑗, 𝑇 denotes the matrix transpose, 𝜇𝑖 is the average rating of item 𝑖, 𝑏𝑖 and 𝑏𝑗 are bias terms for the item and user respectively. To perform the optimization, we need to minimize a loss function such as the regularized squared error: 𝑅=[𝑟11 ⋯ 𝑟1𝑗 ⋮ ⋱ ⋮ 𝑟𝑖1 ⋯ 𝑟𝑖𝑗] min 𝑣∗,𝑢∗∑(𝑟𝑖𝑗−𝜇−𝑏𝑖−𝑏𝑗−𝑣𝑗𝑇𝑢𝑖)2+𝜆(∥𝑣𝑗∥2+∥𝑢𝑖∥2) 𝑅 is a matrix where each element 𝑟𝑖𝑗 indicates the rating of item 𝑖 for user 𝑗 and 𝜆(∥𝑣𝑗∥2+∥𝑢𝑖∥2) is a regularization term based on the L2 norm (other types of regularization can be employed such as L1). 7 Within regression methods, Linear Regression (LinReg) is commonly employed as a baseline model when rating data is available (Mild & Natter, 2002). In the case of recommender systems we need to fit a LinReg model for each item 𝑗 as such: 𝑟𝑖𝑗=𝑣𝑗𝑇𝑢𝑖+𝑏𝑖 The standard method is based on ordinary least squares optimization. As such we can obtain estimates for the 𝑉 parameters using the OLS estimator: 𝑉=(𝑈𝑇𝑈)−1𝑈𝑇𝑅 A key limitation of such methods is the fact that we can only work with rating data (either implicit or explicit). In cases where we only have binary data (for instance, about if a user either downloaded or not a certain app) we require a different modeling approach. The standard method for binary target variables is Logistic Regression (LR), but in this case, since we’re in a multilabel problem, we require Multinomial Logistic Regression (MLR), which can have two forms (Dow & Endersby, 2004): Logit (or Softmax) and Probit. Since logit models are more commonly employed in recommender systems our analysis will focus on these. Let’s suppose we have a set of items 𝒮, a set of users ℛ and a choice matrix 𝐿 such that: 𝐿=[𝑙11 ⋯ 𝑙1𝑏 ⋮ ⋱ ⋮ 𝑙𝑎1 ⋯ 𝑙𝑎𝑏] Where each value of 𝐿 is 1 if a certain user 𝑎∈ℛ chose item 𝑏∈𝒮 and 0 otherwise. Consider now the problem of estimating the probability 𝑃(𝐶𝑟,ℎ) of a new user 𝑟∈ℛ choosing each item ℎ∈𝒮, and that this probability can be estimated from the vector of user attributes 𝑋𝑟=[𝑥𝑟1 ⋮ 𝑥𝑟𝑘]. The multinomial logit model then has the following specification (Aurier & Mejía, 2014): Θ=[𝜃11 ⋯ 𝜃1𝑘 ⋮ ⋱ ⋮ 𝜃ℎ1 ⋯ 𝜃ℎ𝑘] 𝑌𝑟,ℎ=𝑋𝑟[𝜃1 ⋮ 𝜃ℎ]𝑇+𝛼ℎ 𝑃(𝐶𝑟,ℎ)=𝑒𝑌𝑟,ℎ ∑𝑒𝑌𝑟,𝑠 𝑠∈𝒮 Where 𝑌𝑟,ℎ is a measure of the utility of item ℎ for user 𝑟, θ is a matrix of parameters (weights) of each user attribute for each item, and 𝛼ℎ is the constant utility for each item ℎ. We then apply the maximum likelihood estimation method, which is based on cross-entropy loss minimization. The cross-entropy loss function has the following form (Christopher, 2006, p. 209): 𝐸(Θ)=−∑∑𝑙𝑞,𝑤 ln 𝑃(𝐶𝑞,𝑤) 𝑤∈𝒮𝑞∈ℛ 8 A parameter estimate Θ can be obtained in an iterative manner using the Newton-Raphson Method. At each iteration, we take the gradient of the error function in respect to one of the parameters 𝜃𝑝 to update its value (Christopher, 2006, p. 210): ∇𝜃𝑝𝐸(Θ)=∑[𝑃(𝐶𝑞,𝑤)− 𝑞∈ℛ 𝑙𝑞,𝑤]𝑋𝑞 The final recommendations can be found by simply applying the following recommendation function 𝑔 to each value 𝑃(𝐶𝑞,𝑤): 𝑔(𝑥)={1 𝑖𝑓 𝑥>0.5 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 If 𝑔(𝑃(𝐶𝑞,𝑤))=1 we will recommend item 𝑤 for user 𝑞. Several empirical applications of MLR to recommender systems are known (D. M. Fleder & Hosanagar, 2007; S.-H. Yang, Long, Smola, Zha, & Zheng, 2011; Zhao, Zhang, Zhang, & Friedman, 2016). Many other modeling approaches exist for recommender systems. Fischer’s Linear Discriminant Analysis (LDA), also known as Multivariate Discriminant Analysis has been employed for recommender systems (K. Kim, 2011). Tree-based machine learning methods such as Decision Trees (Cho, Kim, & Kim, 2002; Gershman & Meisels, 2010), Random Forests (O’Mahony, Cunningham, & Smyth, 2010; H. R. Zhang & Min, 2016) and Gradient Boosted Regression Trees (Ostuni, Di Noia, Mirizzi, & Di Sciascio, 2014) have also been employed. Bayesian methods were common approaches in the early 2000s (Condli, Madigan, Lewis, & Posse, 1999; Jin & Si, 2004; Miyahara & Pazzani, 2000). Towards the end of the decade some applications of ensemble methods emerged (Jahrer, Töscher, & Legenstein, 2010; Schclar, Tsikinovsky, Rokach, Meisels, & Antwarg, 2009) along with econophysics inspired approaches based on heat and mass diffusion techniques, which have reemerged recently (C. Liu & Zhou, 2010; Lü et al., 2012; Ren, Zhou, & Zhang, 2008; Vidmer, Zeng, Medo, & Zhang, 2015; Y. C. Zhang, Blattner, & Yu, 2007). Kernel methods have also found applications in recommender systems (Abernethy, Bach, Evgeniou, & Vert, 2008; X. Liu et al., 2016) including the use of Support Vector Machines (SVM) (Fortuna, Fortuna, & Mladenić, 2010; Oku, Nakajima, Miyazaki, & Uemura, 2006; Xia, Dong, & Xing, 2006). Kernel methods are based on the kernel trick also known as kernel substitution (Christopher, 2006, p. 292). In these methods we do a mapping of the feature space of new unlabeled inputs 𝑥′ to the training examples 𝑥, using a kernel function 𝑘 that relies on a basis function 𝜑𝑞 such that: 𝑘(𝑥,𝑥′)= 𝜑(𝑥)𝑇𝜑(𝑥′)=∑𝜑𝑞(𝑥)𝜑𝑞(𝑥′) 𝑞∈ℛ The basis function can assume several forms. The most commonly employed form is the Radial Basis Function (RBF) (Christopher, 2006, p. 299). Recently scalable approximation methods based on stochastic methods have emerged, such as Random Fourier Features (RFF) (Rahimi & Recht, 2007). Kernel methods can be seen as a form of instance-based methods, where the kernel function acts as a similarity measure between the training set and the new cases (Christopher, 2006, p. 292). These techniques allow us to linearize the feature space so that simpler decision algorithms can be employed. 9 The application of SVMs to recommender systems involves training an SVM classifier for either each user or each item (Xia et al., 2006). In either case, we are seeking to find a set of items to recommend to each user, by either classifying users for an item or vice-versa. We will assume that a model will be fit for each item, but the inverse specification works in a similar manner. SVMs rely on the kernel mapping to obtain a transformed feature space where a linear decision boundary can be found based on the support vectors, which are sample cases that lie on the maximum margin hyperplanes in the feature space (Christopher, 2006, p. 330). This maximum margin hyperplane for an item 𝑤∈𝒮 can be found by solving (Christopher, 2006, pp. 327–330): arg min Θ,𝑏 {∑𝐸∞([𝜃𝑞𝑘(𝑥,𝑥𝑞)+𝑏]𝑙𝑞,𝑤−1)+𝜆||Θ||2 𝑞∈ℛ } Where 𝐸∞(𝑧) is a function that returns 0 if 𝑧≥0 (which happens when the vector corresponding to 𝑞 is not a support vector), and ∞ otherwise. Alternative multiclass formulations of SVM have been proposed (Hsu & Lin, 2002), which could be used to jointly train a classifier across all items, but no recommender system applications are yet known. Similar methods exist with could potentially be employed to build recommender systems, such as Kernel Logistic Regression (KLR). KLR differs from LR in the definition of 𝑌𝑟,ℎ: 𝑌𝑟,ℎ=𝑘(𝑋,𝑋𝑟)[𝜃1 ⋮ 𝜃ℎ]𝑇+𝛼ℎ A multilabel version of KLR can be achieved by performing the same substitution with softmax instead of LR as in Karsmakers, Pelckmans, & Suykens (2007). KLR has been empirically and analytically demonstrated as having the similar performance and behavior of an SVM (Karsmakers et al., 2007), the main difference being the fact that it requires the entire dataset as opposed to only using the support vectors to build a decision margin (Zhu & Hastie, 2005). As such, it is expected that multiclass KLR should behave similarly to multiclass SVM approaches (Karsmakers et al., 2007). Currently, no applications of KLR (or its multiclass version) are known in recommender systems, but it can potentially improve existing LR/Softmax based methods. Other recommender systems techniques include Artificial Neural Networks (ANN), such as Multilayer Perceptrons (MLP), also known as shallow networks (Goga, Kuyoro, & Goga, 2015). The current state of the art in recommender system design is deep learning methods, which are deep versions of ANN (Cheng et al., 2016; Covington, Adams, & Sargin, 2016; Koutrika, 2018; H. Wang, Wang, & Yeung, 2015; S. Zhang et al., 2017). Both deep and shallow networks can have multiple output nodes that function like either multinomial linear regression or multinomial logistic regression. 2.1.2. Model Evaluation Model evaluation in the context of recommender systems includes several different metrics which range from multiclass/multilabel versions of traditional machine learning and data mining ones to specific profit-centric indicators (Gilotte, Calauzènes, Nedelec, Abraham, & Dollé, 2018; Ju, Choi, Kim, & Moon, 2017). 10 We can divide the evaluation metrics into three main groups: Offline, Online and User Studies (Beel & Langer, 2013). Offline evaluation metrics are the most commonly employed. It assumes that models can be evaluated in terms of prediction/classification accuracy of past item consumption data or ratings. In this context, if an item is recommended to a user who hasn’t previously consumed it, we count it as a model failure. It’s easy to see that offline metrics are not guaranteed to be realistic, since the fact that an item wasn’t previously consumed or rated, doesn’t mean it’s not relevant to that user. Online metrics can overcome these issues by testing the recommender system in a production environment. In this context we can apply several specific recommender system metrics, usually derived from marketing evaluation metrics applied in online advertising and e-commerce (Beel & Langer, 2013). The most common metric is the Click-Through Rate (CTR) (Beel & Langer, 2013): 𝐶𝑇𝑅= 𝐶𝑙𝑖𝑐𝑘𝑠 𝐼𝑚𝑝𝑟𝑒𝑠𝑠𝑖𝑜𝑛𝑠 Where impressions are the number of impressions of all recommended items, and clicks are the number of items that were clicked after being recommended. The assumption of this metric is that a clicked item is relevant to the user. While this assumption is not completely realistic, we can argue that its closer to reality than most offline evaluation metrics. User studies come from the usability and user experience research tradition, which employ surveybased methods. Several standardized psychometric constructs exist to evaluate recommender systems. The ResQues framework proposed by Pu and Chen (2010) includes several constructs that can be measured individually using Factor Analysis or jointly using Structural Equation Modeling (SEM) with either Partial Least Squares (PLS) or Covariance-based estimation (Ayeh, Au, & Law, 2013). These constructs are related to user-perceived qualities, user beliefs, user attitudes and behavioral intentions (Pu & Chen, 2010). Other similar frameworks based on SEM have been proposed such as the Knijnenburg, Willemsen, Gantner, Soncu, & Newell (2012) model which can be used to measure the subjective user experience of a recommender system. These latent psychometric constructs can be used to compare different recommendation models from the point of view of the end-user experience. We will focus on Offline evaluation since that’s the most common approach. Offline evaluation can be thought of as either a classification or a regression (prediction) problem, depending on the type of target variable we have (Herlocker, Konstan, Terveen, & Riedl, 2004). In the former, we are usually working with binary item consumption, while on the latter, we are usually working with explicit or implicit ratings. Regression-type metrics for recommender system evaluation include the Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and Normalized Root Mean Squared Error (NRMSE) (Katsov, 2018, p. 282). In these metrics, we are assuming that the model outputs a prediction 𝑦𝑟,ℎ for actual rating 𝑦𝑟,ℎ by user 𝑟 for item ℎ, resulting in an error term 𝑒𝑟,ℎ such that: 𝑒𝑟,ℎ=𝑦𝑟,ℎ−𝑦𝑟,ℎ MSE has the following definition: 11 𝑀𝑆𝐸=1 |𝑇| ∑ 𝑒2𝑟,ℎ (𝑟,ℎ)∈𝑇 Where 𝑇 is a matrix of unseen user ratings for items used for model testing. MSE is not always convenient because its value cannot be easily compared with the original ratings, since it’s a squared value (Katsov, 2018, p. 282). RMSE allows us to overcome this issue: 𝑅𝑀𝑆𝐸= √𝑀𝑆𝐸 RMSE was the target metric for the Netflix prize (Bennett & Lanning, 2007), and is currently the most popular choice for regression type model evaluation (Szabó, Póczos, & Lőrincz, 2012). NRMSE can now be defined as: 𝑁𝑅𝑀𝑆𝐸= 𝑅𝑀𝑆𝐸 𝑦𝑚𝑎𝑥−𝑦𝑚𝑖𝑛 The advantage of NRMSE is that its value is defined in the range (0,1), which allows us to compare models applied to ratings with different scales. For classification type models we can employ multiclass/multilabel versions of traditional classification metrics, which can usually be done by macro (across classes) or micro (across cases) averaging (Tsoumakas & Vlahavas, 2007). Consider a binary evaluation measure 𝑀(𝑡𝑝,𝑡𝑛,𝑓𝑝,𝑓𝑛) that is calculated based on the number of true positives (𝑡𝑝), true negatives (𝑡𝑛), false positives (𝑓𝑝) and false negatives (𝑓𝑛). Let 𝑡𝑝𝜌, 𝑓𝑝𝜌,𝑡𝑛𝜌 and 𝑓𝑛𝜌 be the number of true positives, false positives, true negatives and false negatives after binary evaluation for a label 𝜌. The macro-averaged and microaveraged versions of 𝑀, are calculated as follows, where 𝐿 is the set of labels (items) (Tsoumakas & Vlahavas, 2007): 𝑀𝑚𝑎𝑐𝑟𝑜=1 |𝐿|∑𝑀(𝑡𝑝𝜌,𝑓𝑝𝜌,𝑡𝑛𝜌,𝑓𝑛𝜌) |𝐿| 𝜌=1 𝑀𝑚𝑖𝑐𝑟𝑜=𝑀(∑𝑡𝑝𝜌 |𝐿| 𝜌=1 ,∑𝑓𝑝𝜌 |𝐿| 𝜌=1 ,∑𝑡𝑛𝜌 |𝐿| 𝜌=1 ,∑𝑓𝑛𝜌 |𝐿| 𝜌=1 ) A popular metric for recommender system evaluation is the F1-Score (Herlocker et al., 2004) which can be defined based on the values of precision and recall: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛= 𝑡𝑝 𝑡𝑝+𝑓𝑝 𝑅𝑒𝑐𝑎𝑙𝑙= 𝑡𝑝 𝑡𝑝+𝑓𝑛 𝐹1 𝑆𝑐𝑜𝑟𝑒=2 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑅𝑒𝑐𝑎𝑙𝑙 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑅𝑒𝑐𝑎𝑙𝑙 Where 𝑡𝑝 is the number of true positives (number of relevant recommendations), 𝑓𝑝 is the number of false positives (number of irrelevant recommendations) and 𝑓𝑛 is the number of false negatives (number of would be relevant items that were not recommended). Logically, and for the sake of completeness, we can further define an additional measure 𝑡𝑛 as the number of true negatives 12 (number of would be irrelevant items that were not recommended). These evaluation metrics usually assume an offline evaluation setting. The most common metric for classification type models is the AUC (Verbeke, Dejaeger, Martens, Hur, & Baesens, 2012): 𝐴𝑈𝐶=∫ 𝐹0(𝑠)𝑓1(𝑠)𝑑𝑠 +∞ −∞ AUC assumes that a classifier produces a score 𝑠=𝑠(𝑥) from the features 𝑥 with a corresponding probability density function of these scores for class 𝑘 instances 𝑓𝑘(𝑠) and cumulative distribution function 𝐹𝑘(𝑠) with two classes 𝑘=0,1. This metric can only be employed to evaluate models that produce probability scores such as softmax regression. Other related metrics are also used including the Gini coefficient (𝐺𝑖𝑛𝑖=2×𝐴𝑈𝐶−1) and the Kolmogorov-Smirnov (KS) statistic which is the maximum distance between a receiver operating curve (ROC) and the diagonal at a specific cut-off value (usually 0.5) (Verbeke et al., 2012). An empirical study by Forman and Scholz (2009) advises the use of “average AUC” (consistent with the previous definition of a Macro measure) and the “F1-Score computed from false and true positives” (consistent with the previous definition of a Micro measure) to compute multilabel versions of AUC and F1-Score respectively. Other popular multiclass classification metrics commonly employed to evaluate recommender systems include the Hamming loss and the Jaccard Index. Hamming loss can be defined as the proportion of misses (Luaces, Díez, Barranquero, del Coz, & Bahamonde, 2012): 𝐻𝑎𝑚𝑚𝑖𝑛𝑔 𝐿𝑜𝑠𝑠= 𝐹𝑃+𝐹𝑁 𝐹𝑃+𝐹𝑁+𝑇𝑃+𝑇𝑁 Jaccard Index can be interpreted as a multilabel measure of accuracy based on the already defined Jaccard Distance (Luaces et al., 2012): 𝐽𝑎𝑐𝑐𝑎𝑟𝑑 𝐼𝑛𝑑𝑒𝑥= 𝑇𝑃 𝐹𝑃+𝐹𝑁+𝑇𝑃 Additional evaluation metrics include Diversity, Coverage, Serendipity and Novelty (Katsov, 2018, pp. 285–288), which originate from desirable recommender system properties (Ge, Delgado-Battenfeld, & Jannach, 2010; Vargas & Castells, 2011). Most of these might be applied in both offline and online settings. We will review some of the most common measurement approaches. Diversity is the ability of the recommender system to produce recommendations that are dissimilar (Katsov, 2018, p. 286). To measure diversity, we need to leverage a content-based similarity metric. One approach is to extract features from the item’s description/title (using text mining and natural language processing techniques), contents (requiring some form of feature extraction from multimedia/hypermedia content), or some other properties. In alternative, we can also obtain a similarity metric between items by factorizing a user by items matrix with the methods we’ve already seen (such as memory-based methods, SVD or even knowledge-based rules). Based on this metric we can define a distance function 𝑑𝑖𝑠𝑡 between items: 13 𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑎,𝑏)∈[0,1] 𝑑𝑖𝑠𝑡(𝑎,𝑏)=𝑑𝑖𝑠𝑡(𝑏,𝑎)=1−𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑎,𝑏) The diversity of the set of recommendations for each user can then be defined as the average distance between all pairs of recommendations. Coverage refers to the percentage of users that the recommender system can give recommendations to (Katsov, 2018, p. 289). This is important because of the sparse nature of the data for recommender systems, which is also the source of the cold start problem (Schein, Popescul, Ungar, & Pennock, 2002). Another view on coverage is related to catalog coverage which is the percentage of the catalog merchandise that is being recommended (Katsov, 2018, p. 289): 𝐶𝑎𝑡𝑎𝑙𝑜𝑔 𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒=1 |𝒮||⋃𝑌𝑢 𝑢∈ℛ | Where 𝑌𝑢 is a recommendation list for user 𝑢, |𝒮| is the cardinality of the set of items 𝒮 and ℛ is the set of all users. Serendipity is a measure of the extent to which recommendations are attractive and surprising (Katsov, 2018, p. 286). Despite being subjective, heuristic approaches have been suggested based on baseline models that produce trivial recommendations (Ge et al., 2010), which are usually simple recommendation system approaches. From that baseline, we will create a set of expected items. Any item that doesn’t belong to this set will be considered unexpected. We can define a usefulness function that returns 1 when a given recommendation is both relevant and unexpected. Serendipity is therefore defined as: 𝑆𝑒𝑟𝑒𝑛𝑑𝑖𝑝𝑖𝑡𝑦=∑𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠(𝑖𝑡𝑒𝑚) 𝑖𝑡𝑒𝑚 ∈ 𝒮 |𝒮| Recommendations are considered novel if the user is not aware of the recommended items at the moment the recommendation is provided (Katsov, 2018, p. 285). Many approaches exist to measure novelty: Popularity-based, Distance-based (Vargas & Castells, 2011) and Time-based (Katsov, 2018, p. 285). Popularity-based novelty measurement takes advantage of the long-tail concept. If a relevant recommended item is less popular, we can assume that it might be more novel. Therefore (Vargas & Castells, 2011): 𝑁𝑜𝑣𝑒𝑙𝑡𝑦𝑃𝑜𝑝𝑢𝑙𝑎𝑟𝑖𝑡𝑦−𝑏𝑎𝑠𝑒𝑑(𝑖𝑡𝑒𝑚,𝑐𝑜𝑛𝑡𝑒𝑥𝑡)=1−𝑝(𝑠𝑒𝑒𝑛|𝑖𝑡𝑒𝑚,𝑐𝑜𝑛𝑡𝑒𝑥𝑡) Distance-based novelty measurement takes advantage of the item features to define a distance measure between items. Then we can leverage the user’s past behavior to measure how novel that item might be for a specific user, by considering its past consumption context. Therefore (Vargas & Castells, 2011): 𝑁𝑜𝑣𝑒𝑙𝑡𝑦𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒−𝑏𝑎𝑠𝑒𝑑(𝑖𝑡𝑒𝑚,𝑢𝑠𝑒𝑟)=min 𝑝𝑎𝑠𝑡 𝑖𝑡𝑒𝑚 ∈ 𝑢𝑠𝑒𝑟[1−𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑖𝑡𝑒𝑚,𝑝𝑎𝑠𝑡 𝑖𝑡𝑒𝑚)] Time-based novelty measurement assumes that the elapsed time between recommendation and action taken on that item by a user indicates its novelty level (Katsov, 2018, p. 285). A larger elapsed time means a higher novelty. Therefore: 14 𝑁𝑜𝑣𝑒𝑙𝑡𝑦𝑇𝑖𝑚𝑒−𝑏𝑎𝑠𝑒𝑑(𝑡𝑖𝑡𝑒𝑚)=𝛾𝑡𝑖𝑡𝑒𝑚 Where 𝑡𝑖𝑡𝑒𝑚 is the elapsed time between the recommendation of the item and the action, and 𝛾 is a time weight parameter. The weight parameter can either be set manually based on the retailer’s experience, or empirically by combining past behavioral data with a user-based novelty psychometric construct (Pu, Chen, & Hu, 2012). We can therefore use a regression model to estimate the parameter, by assuming that the following relationship is true: 𝑁𝑜𝑣𝑒𝑙𝑡𝑦𝑈𝑠𝑒𝑟−𝑏𝑎𝑠𝑒𝑑(𝑖𝑡𝑒𝑚)=𝛾𝑡𝑖𝑡𝑒𝑚 This novelty measurement approach overlaps with the user studies evaluation approach. In ecommerce settings we can also evaluate the recommender system using profit metrics derived from sales: 𝑃𝑟𝑜𝑓𝑖𝑡= ∑𝑄𝑢𝑎𝑛𝑡𝑖𝑡𝑦 𝑆𝑜𝑙𝑑𝑖𝑡𝑒𝑚×𝑀𝑎𝑟𝑔𝑖𝑛𝑖𝑡𝑒𝑚 𝑖𝑡𝑒𝑚𝑠 In environments where we are not selling items (including email, news, multimedia content recommendations, amongst others) this metric cannot be used. In the case of mobile app recommendations, most apps are free to download (94.24% in the Android platform, 88.18% in iOS) (Statista, 2018), which means that this metric can only be used for the small proportion of paid apps. For non-paid items, we need to consider a different approach to profit measurement based on the impact of the recommender system on the Customer Lifetime Value (𝐶𝐿𝑉) (Iwata, Saito, & Yamada, 2008). CLV is commonly used to guide Customer Relationship Management (CRM) processes (Blattberg, Kim, & Neslin, 2008, p. 163). The revenue associated with each successfully recommended item is the portion of the 𝐶𝐿𝑉 that can be attributed to that item. Assuming that the revenue associated with each item is constant for all items, then the portion of the 𝐶𝐿𝑉 attributed to each item can be derived from the item consumption probability (Iwata et al., 2008). In environments such as mobile app stores, users can perform several different actions, but all of these are ultimately related with app consumptions, therefore its plausible to assume that app acquisitions are the major component of the 𝐶𝐿𝑉, and that these are heavily influenced by recommender systems. 𝐶𝐿𝑉 can be calculated using several methods. Traditionally 𝐶𝐿𝑉 would be estimated using a Recency, Frequency and Monetary (RFM) model to rank users (Gupta et al., 2006). Modern approaches extend the concept of RFM to obtain a financial estimation of the 𝐶𝐿𝑉 for each customer. One approach to obtaining a global average of 𝐶𝐿𝑉 is to calculate the average of all individual-level 𝐶𝐿𝑉 values across a database, where the 𝐶𝐿𝑉 for each customer is given by (Blattberg et al., 2008, p. 108): 𝐶𝐿𝑉=∑(𝐷𝑡−𝐶𝑡)𝑆𝑡 (1+𝛿)𝑡−1 ∞ 𝑡=1 Where 𝛿 is the discount rate per time unit 𝑡, 𝐷𝑡 is the revenue generated by the user on moment 𝑡, 𝐶𝑡 is the cost of serving user on moment 𝑡, 𝑆𝑡 is the probability of the customer not churning before 15 moment 𝑡. The RFM model is incorporated into 𝐶𝐿𝑉 through the 𝑆𝑡 (“recency” and “frequency”) and 𝐷𝑡 (“monetary”). Several methods to obtain 𝐷𝑡 have been proposed (Blattberg et al., 2008, pp. 130–131): The simplest approach is to assume that 𝐷𝑡 is constant in all periods based on the individual average or the global Average Revenue Per User (ARPU). Trend, causal (based on the user features) and stochastic models are also commonly employed. To model 𝑆𝑡=𝑝(𝑎𝑙𝑖𝑣𝑒) we need a consumer behavior model. Several have been proposed: Beta Binomial/NBD (BB-NBD) (Jeuland, Bass, & Wright, 1980), NBD-Dirichlet (Goodhardt, Ehrenberg, & Chatfield, 1984), Pareto/NBD (P-NBD) (Schmittlein, Morrison, & Colombo, 1987), BetaGeometric/Beta-Binomial (BG-BB) (Fader, Hardie, & Berger, 2004), and Beta-Geometric/NBD (BG-NDB) (Fader, Hardie, & Lee, 2005). We will focus our attention on the most recent model by Fader et al. (2005), according to which 𝑝(𝑎𝑙𝑖𝑣𝑒) is given by (Fader & Hardie, 2008): 𝑝(𝑎𝑙𝑖𝑣𝑒|𝑥,𝑡𝑥,𝑇,𝑟,𝛼,𝑎,𝑏)=1 1+ 𝑎 𝑏+𝑥(𝛼+𝑇 𝛼+𝑡𝑥)𝑟+𝑥 Where 𝑥 is the number of transactions observed in the time-period (0,𝑇] (“frequency”) and 𝑡𝑥(0 < 𝑡𝑥 ≤ 𝑇) is the time of the last transaction (“recency”). The model’s four parameters 𝑟,𝛼,𝑎,𝑏 can be estimated using maximum likelihood estimation from the likelihood function: 𝐿(𝑟,𝛼,𝑎,𝑏 |𝑋 = 𝑥,𝑡𝑥,𝑇) =𝐵(𝑎,𝑏 + 𝑥) 𝐵(𝑎,𝑏)𝛤(𝑟 + 𝑥)𝛼𝑟 𝛤(𝑟)(𝛼 + 𝑇)𝑟+𝑥 + 𝛿𝑥>0𝐵(𝑎 + 1,𝑏 + 𝑥 − 1) 𝐵(𝑎,𝑏) 𝛤(𝑟 + 𝑥)𝛼𝑟 𝛤(𝑟)(𝛼 + 𝑡𝑥)𝑟+𝑥 Suppose we have a sample of 𝑁 customers, where customer 𝑖 had 𝑋𝑖 = 𝑥𝑖 transactions in the period (0,𝑇𝑖], with the last transaction occurring at 𝑡𝑥𝑖. The sample log-likelihood function is: 𝐿𝐿(𝑟,𝛼,𝑎,𝑏)=∑𝑙𝑛 𝐿(𝑟,𝛼,𝑎,𝑏 |𝑋𝑖 = 𝑥𝑖,𝑡𝑥𝑖 ,𝑇𝑖) 𝑁 𝑖=1 By maximizing this function using standard optimization methods, we can obtain the parameter estimates. While BG/NBD model and its extensions are the current state of the art, it requires individual-level data to estimate the parameters. The already mentioned NBD-Dirichlet model (Goodhardt et al., 1984) can be an alternative to this model with simpler data requirements. NBD-Dirichlet results from the combination of two distributions: the Negative Binomial Distribution (NBD) and the Dirichlet Multinomial Distribution (DMD) (Dawes, Meyer-Waarden, & Driesener, 2015). The NBD part describes the category buying behavior of individuals in a market, while the DMD part models the probability of each individual in the market purchasing a specific brand (Goodhardt et al., 1984). The resulting model is therefore given by (Goodhardt et al., 1984): 𝑝(𝑟𝑗|𝑛)=(𝑛 𝑟𝑗)Β(𝛼𝑗+𝑟𝑗,𝑆−𝛼𝑗+𝑛−𝑟𝑗) Β(α𝑗,𝑆−α𝑗) 22 3. METHODOLOGY To answer the research questions of this study, an empirical study was conducted on a Portuguese Android app store, Aptoide. This platform is presented as a “social app store” where some common online social network features exist. Users can create their profiles and follow each other, they have access to a microblogging feature (through the “apps timeline” feature which was still active at the time of the dataset extraction), user-generated comments and app reviews. Additionally, users can download and share apps with their followers. It also has a search engine to find Android apps. Several different models will be compared using a novel profit-centric metric. The different deep learning models compared were defined to answer the three specific research questions. This means that we will do model comparisons across three levels: 1. Feature Embedding, 2. Data Sources and 3. Scoring. Figure 4 – Recommendations displayed in the Aptoide mobile app home. 23 3.1. PROPOSED MODEL We propose a generic architecture for mobile app recommendations. The basic architecture of our model has three basic layers: feature embedding, feature extraction, and scoring. The proposed model is a fully connected deep network designed to score mobile apps for users using aggregated features at the user level. The raw data can include behavioral, social-network and user-generated content data. Figure 5 – Deep-Learning Architecture Overview The three first layers are fully connected components with Relu activation functions. The embedding layer is pre-trained using a combined Word2Vec and Autoencoder procedure from both numerical raw features and one-hot encoded categorical variables. The resulting embeddings are fed into the following layer. The feature extraction layers take the embedded features for each user in each moment and performs a dimensionality reduction. These features will be used as input to the final scoring layer. Two variants of the architecture can be constructed using different scoring methods. The standard approach is to employ Softmax. As a novel approach to recommender system design, we propose an alternative multiclass classification method based on Kernel methods. For this purpose, we will use Tensorflow’s implementation of Kernel methods, which is based on Random Fourier Features (RFF) (Rahimi & Recht, 2007). By combining RFF with a softmax output node, we can implement Multiclass Kernel Logistic Regression (M-KLR) (Karsmakers et al., 2007). Tensorflow doesn’t currently offer a differentiable SVM or multiclass SVM implementation, but Kernel Logistic Regression (KLR) has been empirically and analytically demonstrated as having the similar performance and behavior as SVM (Karsmakers et al., 2007), the main difference being the fact that it requires the entire dataset as opposed to only using support vectors to build a decision margin (Zhu & Hastie, 2005). As such, it is expected that M-KLR should behave similarly to multiclass SVM approaches. Scoring Method Scores Scoring 24 This proposed architecture can serve as the foundation for the research design described below, where different variants of this architecture will be empirically tested. 3.2. EXPERIMENTAL SETUP Three specific research questions can be formulated to satisfy the three specific research objectives previously presented: 1. What is the impact of using unstructured data versus only using structured data in the model performance? 2. What is the impact of embedding layer pre-training in the model performance? 3. How efficient are Kernel-based methods in the scoring layer? The basic architecture to be tested is the recommender system architecture previously described. Different components will be omitted across models to answer each research question. The following table presents a description of the four models: Model 1 Three-layer network with fully connected (FC) feature embedding layer (numeric features), feature extraction layer, softmax output and no pretraining. Model 2 Four-layer network with one layer of FC feature embedding (numeric and categorical features) and two feature extraction layers, softmax output and no pretraining. Model 3 Four-layer network with one FC feature embedding (numeric and categorical features) and two feature extraction layers, softmax output and embedding pretraining. Model 4 Four-layer network with one FC feature embedding (numeric and categorical features) and two feature extraction layers, kernel softmax output and embedding pretraining. The task in our experiment consists of scoring a batch of 10 apps for each user as a multilabel classification problem. We recommend an app if the score is over a threshold (commonly 0.5). If the user has previously acquired that app we consider it a true positive. If not, we consider it a false positive. The same logic is applied to negatives. Our research questions will be answered by the results of each model on this task. To answer research question 1, we will compare the performance of models 1 and 2. To answer research question 2, we will compare the performance of models 2 and 3. To answer research question 3 we will compare the performance of models 3 and 4. The Python source code for the experiment is available at https://github.com/lgpintomkt/MasterThesis. 25 Figure 8 – Architecture of Model 4 Scores Scores S Scores S K S S Softmax Kernel Mapping K Fully Connected Scoring Scoring Scoring 20 units 20 units 659 features 30 units 20 units 60 units Kernel Size = 3000 10 units 10 units 10 units 172 features 30 units 659 features 30 units 60 units Figure 7 – Architecture of Model 1 Figure 6 – Architecture of Model 2 and 3 26 3.3. MODEL EVALUATION Our base model is a classification style model that provides probability scores for items. The different variations will be compared in an offline setting. Therefore, standard recommender system metrics such as Hamming Loss and Jaccard Index along with averaged versions of AUC, Precision, Recall and F1-Score can be employed to compare model performance. In the case of AUC, we will employ Macro averaging, while in the case of F1-Score, Precision and Recall we will use Micro-averaging consistent with the results of Forman and Scholz (2009). An additional Macro average F1-Score metric will also be included for comparison. We propose an additional novel offline profit-based metric based on the concept of the estimated Average App Download Value (𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑) which is derived from CLV metrics and the NBD-Dirichlet model (full mathematical derivation included in the Appendix A). We considered that any true positive results in an increment of +𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑 (revenue) while any false positive is counted as decrement of −𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑 (opportunity cost). Additional cost variables were added to compare the efficiency of each models considering the model training, validation, testing and deployment: 𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑=∑𝑛𝐷(𝑅𝑡−𝐶𝑡)[Β(α𝑗,𝑆−α𝑗)−Β(𝛼𝑗,𝑆−𝛼𝑗+𝑛)]Β(𝛼𝑗+1,𝑆−𝛼𝑗+𝑛−1) (1+𝛿)𝑡−1Β(α𝑗,𝑆−α𝑗)2 ℎ 𝑡=1 𝑃𝑟𝑜𝑓𝑖𝑡𝑚𝑜𝑑𝑒𝑙=(𝑈𝜏𝑚𝑜𝑑𝑒𝑙−𝑈𝜑𝑚𝑜𝑑𝑒𝑙)𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑−𝑡𝑈𝐶𝑜𝑠𝑡𝑚𝑜𝑑𝑒𝑙 𝑑𝑒𝑝𝑙𝑜𝑦𝑚𝑒𝑛𝑡−𝐶𝑜𝑠𝑡𝑡𝑟𝑎𝑖𝑛𝑖𝑛𝑔 𝑣𝑎𝑙𝑖𝑑𝑎𝑡𝑖𝑜𝑛 𝑡𝑒𝑠𝑡𝑖𝑛𝑔 The value for 𝑈 corresponds to the size of the testing set. The parameters 𝛼,𝐷,𝑅𝑡,𝐶𝑡,𝛿 and ℎ were obtained from internal Aptoide data. The values for 𝑛=13.52 and 𝑆=29.88 were estimated from recent public industry reports (App Annie, 2018). The values for 𝜏𝑚𝑜𝑑𝑒𝑙 (true positive rate) and 𝜑𝑚𝑜𝑑𝑒𝑙 (false positive rate) come from the performance of each model during the testing stage. The values for 𝐶𝑜𝑠𝑡𝑚𝑜𝑑𝑒𝑙 𝑑𝑒𝑝𝑙𝑜𝑦𝑚𝑒𝑛𝑡 and 𝐶𝑜𝑠𝑡𝑡𝑟𝑎𝑖𝑛𝑖𝑛𝑔 𝑣𝑎𝑙𝑖𝑑𝑎𝑡𝑖𝑜𝑛 𝑡𝑒𝑠𝑡𝑖𝑛𝑔 come from the cloud platforms hourly costs that will be discussed in the following sections. The deployed model execution time (𝑡) will be estimated using a separate experiment conducted on a Spark environment. In this case, we are interested in the expected execution time for the entire Spark job associated with the model deployment. Remember that Spark jobs are made of multiple stages, and each stage contains several tasks running in parallel in a distributed manner (K. Wang & Khan, 2015). The Tensorflow model to be deployed is encapsulated in an object that will require only a single stage, with several tasks. The total job execution time is therefore given by the following expressions (K. Wang & Khan, 2015): 𝑃=∑𝐶𝑜𝑟𝑒𝑁𝑢𝑚𝑖 𝐻 𝑖=1 𝐽𝑜𝑏𝑇𝑖𝑚𝑒 = 𝐽𝑜𝑏𝑆𝑡𝑎𝑟𝑡𝑢𝑝 + 𝑆𝑡𝑎𝑔𝑒𝑇𝑖𝑚𝑒 + 𝐽𝑜𝑏𝐶𝑙𝑒𝑎𝑛𝑢𝑝 27 𝑆𝑡𝑎𝑔𝑒𝑇𝑖𝑚𝑒=𝑆𝑡𝑎𝑔𝑒𝑆𝑡𝑎𝑟𝑡𝑢𝑝+𝑚𝑎𝑥𝑣=1 𝑃∑𝑇𝑎𝑠𝑘𝑇𝑖𝑚𝑒𝑣,𝑖 𝑅𝑣 𝑖=1 + 𝑆𝑡𝑎𝑔𝑒𝐶𝑙𝑒𝑎𝑛𝑢𝑝 Where 𝐶𝑜𝑟𝑒𝑁𝑢𝑚 is the number of CPU cores of working node 𝑖 and 𝐻 is the number of working nodes in the cluster, 𝑅𝑣 is the number of sequential tasks executed on CPU core 𝑣 and 𝑃 is the total number of CPU cores in the Spark cluster. The number of sequential tasks 𝑅𝑣 under these conditions will be the number of user batches (RDD partitions) included in the testing experiment divided by the number of CPU cores. We can separate the total job time in two components as such: 𝑍=𝐽𝑜𝑏𝑆𝑡𝑎𝑟𝑡𝑢𝑝+𝑆𝑡𝑎𝑔𝑒𝑆𝑡𝑎𝑟𝑡𝑢𝑝+𝑆𝑡𝑎𝑔𝑒𝐶𝑙𝑒𝑎𝑛𝑢𝑝+𝐽𝑜𝑏𝐶𝑙𝑒𝑎𝑛𝑢𝑝 𝑉=𝑚𝑎𝑥𝑣=1 𝑃∑𝑇𝑎𝑠𝑘𝑇𝑖𝑚𝑒𝑣,𝑖 𝑅𝑣 𝑖=1 𝐽𝑜𝑏𝑇𝑖𝑚𝑒=𝑍+ 𝑉 Since this experiment is to be executed in a single machine cluster we will simulate the typical industrial cluster configurations, by estimating the 𝑉 and 𝑍 components using Ψ sequential trials (the number of trials meant to simulate the number of nodes in the cluster). To simplify the calculations, we will employ a single processing CPU core. We can define two finite sets 𝒱={v1,…vΨ} and 𝒵={z1,…zΨ} which represent the Ψ observations of 𝑍 and 𝑉. Each element of 𝒱 represents the sum of all task durations in the trial. From these, we can define two estimators for the actual values. For 𝑍 we are looking for the expected value 𝜇(𝑍). For 𝑉 we are looking for the maximum expected value 𝜇∗(𝑉), which can be estimated using the maximum estimator (ME) (van Hasselt, 2013). Therefore, we have: 𝑉 ≈ 𝑉≡𝜇∗(𝑉) ≡ max (𝒱) 𝑍≈ 𝑍󰆹≡𝜇(𝑍)≡∑𝒵 |𝒵| The final 𝑡 will then be given by: 𝑡=max (𝒱) + ∑𝒵 |𝒵| To simulate the typical Big Data setups, we will assume Ψ=12. This cluster size was the same used by K. Wang & Khan (2015) in their experiment, and consistent with the reported industry best practices (Fujitsu, 2017). Even though much larger cluster sizes may occur for specific tasks (Apache Foundation, 2018a), these do not appear to be common. The runtimes were extracted from the Spark Web UI which displays runtimes for tasks, stages, and jobs. We will assume 𝜉=2.958 𝑈𝑆𝐷/ℎ𝑜𝑢𝑟 since it’s the current pricing of a general purpose m5 12 node cluster on the most popular infrastructure provider (AWS), which includes both the EC2 and EMR costs (Amazon Web Services, 2018). 28 3.4. DATASET DETAILS 3.4.1. Raw Data The dataset employed to train and test the different models was extracted from the Aptoide’s Big Data lake running on Amazon Web Services (AWS). Table 1 below lists the features available on the dataset. Table 1 – Features Geographic2 Demographic Technographic Behavioral Social Latitude Longitude Language Android Version Device Model # Downloads # Searches # Clicks App Downloads Search Queries Clicked Items Social Network Topology 3.4.2. Feature Engineering As previously stated, the raw features were subjected to a feature engineering process which will be described in the following section. The geographic features were stored as latitude/longitude coordinates (in degrees) for each user, however, these cannot be fed directly to a deep learning model. A method similar to Locally Linear Embedding with Geodesic Distance (Varini, Degenhard, & Nattkemper, 2005), based on classical Multidimensional Scaling (MDS) and KNN was employed to extract usable features. The first step was to convert the coordinates in degrees to radians. Once in radian format, a random subsample of 40.000 users was taken from the original sample. From these, a validation and test subsample were also extracted, each being 15% of the overall subsample, with the remaining 70% were used for training. A distance matrix between all users on the subsample was computed. The employed distance metric was the implementation of the Karney geodesic distance (2013) available on the Python geocoding library GeoPy. Karney’s method is currently the state of the art in geodesy, and its widely used to estimate the distance between two latitude/longitude pairs, building upon earlier methods such as Vincenty’s formulae (1975). The next step was the application of Singular Value Decomposition (SVD) to the distance matrix, where two eigenvectors were retained. Finally, the embedding was extended to the entire dataset using KNN regression for each eigenvector, where the best K parameter was found to be 5. To assess goodness of fit, the average R2 of both dimensions was used to evaluate distance reconstruction quality on the test set. The final R2 value was 0.989, indicating a good feature embedding accuracy. This very high goodness of fit might be explained by having only city-level geographic coordinates, resulting in a perfect coordinate overlap between nearby users. For the demographic and technographic features, the top 99 most frequent occurrences for each variable were extracted. A 100th feature was included to represent the “other” hypothesis. A one-hot encoded vector was computed for each user. The behavioral features included clickstream/touchstream variables, search queries, and app download data. From these, app download data were categorical (the user either downloaded or not 2 The coordinates for each user are the coordinates of the centre of each user’s first login city. 29 a certain app), and the same procedure of the demographic/technographic features was applied. The clickstream data was encoded in a manner consistent with the tidy data approach (Wickham, 2014), where a column encoded the number of clicks/touches of each user, and other columns encoded the percentage of those clicks that were of a certain category/type. The search query data was subjected to a more complex procedure. The objective was to obtain a categorical one-hot encoding. However, such an encoding should reflect the similarity of mistyped terms or slightly different terms that may refer to the same intent. The approach we followed was derived from standard procedures applied in Natural Language Processing (NLP) based on spellchecking dictionaries (Lehal, 2007). The approach here relied on building a dictionary that would be used to perform one-hot encoding of the search terms. To build this dictionary a sample of the most common 10.000 terms was extracted. A distance matrix was computed for all terms using DamerauLevenshtein distance. An MDS procedure was then applied to this matrix, where two dimensions were retained. Using these dimensions, a hierarchical clustering procedure was applied, using a height cutoff of 12, resulting in a total of 100 clusters. The final dictionary was built by extracting the most common term from each cluster. The spell-checking procedure was applied to the entire data set by replacing each term with the most similar term according to the Damerau-Levenshtein distance. The search terms employed by the users were then one-hot encoded using the dictionary entries. The feature extraction for the social network topology data was done using a procedure derived from the DeepWalk method already described. (Perozzi et al., 2014). A random walking was done on this network, with walk length 10, and 0% restarting probability. The resulting random walks allowed us to generate a one-hot encoded vector for each user based on the top 100 most popular profile IDs following a bag-of-words logic. Two centrality measures were also computed and added to the final features: indegree and outdegree. These two features correspond to the number of followers and followed respectively, and capture some additional global network properties that are not so well captured by the random walking procedure (Cui, Wang, Pei, & Zhu, 2017; Dalmia & Gupta, 2018). The pre-training procedure for the network was the naïve method described in the CENE experiments (Sun et al., 2016) since it was reported to give good results. The same logic of CENE was followed, where we assumed to be observing a directed network where each user is associated not only with other user profiles but also apps, search queries, device models, Android versions, and languages. Ngrams were generated for each user from the one-hot encoded variables. The embedding of this network is then achieved using Word2Vec for the categorical variables and an Autoencoder implemented in Tensorflow for the numerical variables. Finally, every feature was normalized using min-max normalization resulting in values between 0 and 1. The total number of extracted features amounts to 659, of which 172 are numerical, the remaining ones being categorical (one-hot encoded). Users with less than 97% sparsity were kept. The final data is represented as a rank-3 tensor 𝑊 such that each element 𝑊𝑎,𝑏,𝑐 corresponds to the value of feature 𝑎 for user 𝑏 on day 𝑐, over the first 30 days of using the platform. From this tensor, two different matrixes were then extracted (one with just the 172 numerical features and another one with the full 659 feature-set), where the values over the 30 days are summarized for each user. These condensed datasets are going to be the inputs for our models. The pre-processing procedure was executed in a local AMD Quad-Core Processor A4-5000 (1.5 GHz) CPU machine, lasting approximately 8 days. 30 4. RESULTS The processed dataset was divided into three parts making up the training, validation and test sets, roughly corresponding to 86%, 2% and 12% of the overall data (74.209 users). The model training, validation, and testing were done on a Nvidia Tesla K80 GPU instance executing a Tensorflow environment using the FloydHub service (https://www.floydhub.com/). 4.1. MODEL TRAINING AND VALIDATION The training of models 3 and 4 involved a pretraining of the feature embedding architectural component using a Word2Vec (for the unstructured data) and Autoencoder implementation in Tensorflow. Both models were trained over 10 epochs with gradient descent. The resulting embedding parameters where concatenated (cross-loadings were randomly initialized with lower values close to zero) and used as the initialization tensor for the first layer of the final architecture. This pretraining was followed by a global training stage. The training set was divided into 32 batches of 2.000 users (64.000 overall). The training procedure was done under the classic early stopping method (Prechelt, 2012) dependent on the loss on the entire validation set computed at each epoch (if the validation loss increases on this epoch, the training stops, and the previous weights are retained). The Adam optimizer with the standard parameters (Kingma & Ba, 2014) was used to perform the network training backpropagation. 0 1 2 3 4 5 1 2 3 Loss Epoch Model 2 (Training) Model 2 (Validation) 2.6 2.7 2.8 2.9 3 3.1 3.2 3.3 12345678910 11 12 13 Loss Epoch Model 1 (Training) Model 1 (Validation) 0 0.2 0.4 0.6 0.8 1 1357911 13 15 17 19 21 23 25 27 Loss Epoch Model 4 (Training) Model 4 (Validation) 0.8 0.85 0.9 0.95 1 1.05 1357911 13 15 17 19 21 23 25 27 29 31 33 Loss Epoch Model 3 (Training) Model 3 (Validation) Chart 3 – Training and Validation Loss (Model 1) Chart 2 - Training and Validation Loss (Model 2) Chart 5 - Training and Validation Loss (Model 3) Chart 4 - Training and Validation Loss (Model 4) 31 4.2. MODEL TESTING The model testing was done in two parts. The first part was meant to test the model accuracy using standard metrics on the testing set on the Nvidia Tesla K80 GPU instance where the training and validation were done. The average pricing per hour of the infrastructure provider (FloydHub) was used to compute the total costs of training, validation, and testing (1.22 USD/hour). Table 2 – Computational Statistics of Model Training, Validation and Testing. Model 1 Model 2 Model 3 Model 4 Max CPU Utilization 56% 72% 74% 55% Max GPU Utilization 5% 0% 6% 27% Execution Time (minutes:seconds) 03:34 03:22 05:35 04:38 Total Cost of Training, Validation and Testing (USD) $0.071 $0.067 $0.111 $0.093 A portion of the testing set (2250 users) was then allocated to a secondary test to obtain the Spark execution time component. Table 3 – Computational Statistics of Model Deployment Model 1 Model 2 Model 3 Model 4 Estimated Serving Latency (ms) 7.56 8.45 15.11 15.34 Cost per User $ 0.0003104 $0.0003470 $0.0006209 $0.0006304 Total Cost of Deployment (USD) $ 0.70 $0.78 $1.40 $1.42 The Spark estimated serving latency was measured separately on an AMD Quad-Core Processor A45000 (1.5 GHz) CPU machine executing a Spark server on an Ubuntu VM through a Windows 10 host. A single CPU core of this machine was used to execute the experiment. The obtained execution time for each model was used to estimate the Information Technology (IT) infrastructure cost component. A Spark script was executed directly on the PySpark command line tool, which imports the Tensorflow graph from the local disk before applying it to each row of an RDD, according to the method proposed by Databricks (Hunter, 2016). The values from both tests were combined to compute the final Profit value for each model. Hamming Loss, Jaccard Index, AUC (Macro), F1 Score (Macro and Micro), along with Precision and Recall were also compared across models. Chart 6 – Experimental Results of Model Testing 0% 20% 40% 60% 80% 100% Profit (Normalized) 1 - Hamming Loss Jaccard Index AUC (Macro) F1 Score (Macro) F1 Score (Micro) Precision Recall Model 1 Model 2 Model 3 Model 4 38 us greater confidence on the quality of the results we’ve obtained, while also contributing to the nascent academic research in the field of distributed GP-GPU and the efficient deployment of deeplearning models at Big Data scale. 39 REFERENCES Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., … Zheng, X. (2016). TensorFlow: LargeScale Machine Learning on Heterogeneous Distributed Systems. https://doi.org/10.1109/TIP.2003.819861. Abdullah, A., Veltkamp, R. C., & Wiering, M. A. (2009). An ensemble of deep support vector machines for image categorization. SoCPaR 2009 - Soft Computing and Pattern Recognition, (1), 301–306. https://doi.org/10.1109/SoCPaR.2009.67 Abernethy, J., Bach, F., Evgeniou, T., & Vert, J.-P. (2008). A new approach to collaborative filtering: operator estimation with spectral regularization. The Journal of Machine Learning Research, 10, 803–826. https://doi.org/10.1145/1577069.1577098 Aha, D. W., Kibler, D., & Albert, M. K. (1991). Instance-Based Learning Algorithms. Machine Learning, 6(1), 37–66. https://doi.org/10.1023/A:1022689900470 Aljukhadar, M., Senecal, S., & Daoust, C. E. (2010). Information overload and usage of recommendations. In CEUR Workshop Proceedings (Vol. 612, pp. 26–33). Alsheikh, M. A., Niyato, D., Lin, S., Tan, H.-P., & Han, Z. (2016). Mobile big data analytics using deep learning and apache spark. IEEE Network, 30(3), 22–29. https://doi.org/10.1109/MNET.2016.7474340 Amadeo, R. (2016). The Google Play Store scraps Google+ integration. Retrieved July 29, 2018, from https://arstechnica.com/gadgets/2016/08/the-google-play-store-scraps-google-integration/ Amatriain, X. (2013). Mining large streams of user data for personalized recommendations. ACM SIGKDD Explorations Newsletter, 14(2), 37. https://doi.org/10.1145/2481244.2481250 Amazon Web Services. (2018). Amazon EMR Pricing. Retrieved June 10, 2018, from https://aws.amazon.com/emr/pricing/?nc1=h_ls Anderson, C. (2008). The Long Tail, Revised and Updated Edition: Why the Future of Business is Selling Less of More. Word Journal Of The International Linguistic Association. https://doi.org/30 Apache. (2017). Feature Extraction and Transformation - RDD-based API. Retrieved June 8, 2017, from https://spark.apache.org/docs/2.1.0/mllib-feature-extraction.html#word2vec Apache Foundation. (2018a). Apache Spark FAQ. Retrieved June 10, 2018, from https://spark.apache.org/faq.html Apache Foundation. (2018b). Dimensionality Reduction - RDD-based API. Retrieved August 9, 2018, from https://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html#dimensionalityreduction-rdd-based-api Apache Foundation. (2018c). Feature Extraction and Transformation - RDD-based API. Retrieved August 9, 2018, from https://spark.apache.org/docs/2.2.0/mllib-feature-extraction.html App Annie. (2017). App Annie App Monetization Report 2016. Retrieved from http://files.appannie.com.s3.amazonaws.com/reports/1116_App_Monentization_Report_EN.p df?mkt_tok=eyJpIjoiWXpSaE5UQXlZemcxWm1NMCIsInQiOiJxZE1Dd0J0RnRiOWFUamJaeEc0YkF qZEpPQkNEc3JGZGtjYVNPS1ZseGNlckRTb2FjWTM3UVR3ejRqbnR1bzRtMnFBbkozWmNcL2Z0d2 9BNUVzcVB2cUtRaHV 40 App Annie. (2018). App Annie 2017 Retrospective. App Annie. Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., … Lacoste-Julien, S. (2017). A Closer Look at Memorization in Deep Networks. Retrieved from http://arxiv.org/abs/1706.05394 Aurier, P., & Mejía, V. (2014). Multivariate Logit and Probit models for simultaneous purchases: Presentation, uses, appeal and limitations. Recherche et Applications En Marketing (English Edition), 29(2), 75–94. https://doi.org/10.1177/2051570714535531 Ayeh, J. K., Au, N., & Law, R. (2013). “Do We Believe in TripAdvisor?” Examining Credibility Perceptions and Online Travelers’ Attitude toward Using User-Generated Content. Journal of Travel Research, 52(4), 437–452. https://doi.org/10.1177/0047287512475217 Barragáns-Martínez, A. B., Costa-Montenegro, E., Burguillo, J. C., Rey-López, M., Mikic-Fonte, F. A., & Peleteiro, A. (2010). A hybrid content-based and item-based collaborative filtering approach to recommend TV programs enhanced with singular value decomposition. Information Sciences, 180(22), 4290–4311. https://doi.org/10.1016/j.ins.2010.07.024 Batselier, J., & Vanhoucke, M. (2017). Improving project forecast accuracy by integrating earned value management with exponential smoothing and reference class forecasting. International Journal of Project Management, 35(1), 28–43. https://doi.org/10.1016/j.ijproman.2016.10.003 Beel, J., & Langer, S. (2013). A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems, xx(x). https://doi.org/10.1007/978-3319-24592-8_12 Bennett, J., & Lanning, S. (2007). The Netflix Prize. KDD Cup and Workshop, 3–6. https://doi.org/10.1145/1562764.1562769 Bilgihan, A., Kandampully, J., & Zhang, T. (Christina). (2016). Towards a unified customer experience in online shopping environments: Antecedents and outcomes. International Journal of Quality and Service Sciences, 8(1), 102–119. https://doi.org/10.1108/IJQSS-07-2015-0054 Blattberg, R. C., Kim, B.-D., & Neslin, S. A. (2008). Database Marketing - Analyzing and Managing Customers. Springer Berlin Heidelberg. Bloch, F., Jackson, M. O., & Tebaldi, P. (2016). Centrality Measures in Networks. https://doi.org/10.2139/ssrn.2749124 Bobadilla, J., Ortega, F., Hernando, A., & Gutiérrez, A. (2013). Recommender systems survey. Knowledge-Based Systems, 46, 109–132. https://doi.org/10.1016/j.knosys.2013.03.012 Bosagh Zadeh, R., Meng, X., Ulanov, A., Yavuz, B., Pu, L., Venkataraman, S., … Zaharia, M. (2016). Matrix Computations and Optimization in Apache Spark. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16 (pp. 31–38). New York, New York, USA: ACM Press. https://doi.org/10.1145/2939672.2939675 Bound, J. A. (2009). The S Parameter in the Dirichlet-NBD Model : a Simple Interpretation. Journal of Empirical Generalisations in Marketing Science, 12(3), 1–7. Retrieved from https://www.empgens.com/article/the-s-parameter-in-the-dirichlet-nbd-model-a-simpleinterpretation/ boyd, danah m., & Ellison, N. B. (2007). Social Network Sites: Definition, History, and Scholarship. Journal of Computer-Mediated Communication, 13(1), 210–230. https://doi.org/10.1111/j.10836101.2007.00393.x 41 Brand, M. (2003). Fast online SVD revisions for lightweight recommender systems. Proceedings of the 2003 SIAM International Conference on Data Mining, 35(TR-2003-14), 37–46. https://doi.org/10.1137/1.9781611972733.4 Brynjolfsson, E., Hu, Y. (Jeffrey), & Simester, D. (2011). Goodbye Pareto Principle, Hello Long Tail: The Effect of Search Costs on the Concentration of Product Sales. Management Science, 57(8), 1373– 1386. https://doi.org/10.1287/mnsc.1110.1371 Brynjolfsson, E., Hu, Y. J., & Smith, M. D. (2006). From Niches to Riches: Anatomy of the Long Tail. Sloan Management Review, 47(4), 67. https://doi.org/10.2139/ssrn.918142 Burke, R. (2000). Knowledge-Based Recommender Systems. In Encyclopedia of Library and Information Science: Volume 69 - Supplement 32 (pp. 167–197). https://doi.org/10.1007/978-3-319-296593_5 Bytheway, A. (2014). Investing in Information - The Information Management Body of Knowledge. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-11909-0 Campos, V., Sastre, F., Yagües, M., Bellver, M., Giró-i-Nieto, X., & Torres, J. (2017). Distributed training strategies for a computer vision deep learning algorithm on a distributed GPU cluster. Procedia Computer Science, 108, 315–324. https://doi.org/10.1016/j.procs.2017.05.074 Chen, F. (2016). NBD-Dirichlet Model of Consumer Buying Behavior for Marketing Research. CRAN. Retrieved from https://cran.r-project.org/web/packages/NBDdirichlet/NBDdirichlet.pdf Chen, M., Mao, S., & Liu, Y. (2014). Big data: A survey. Mobile Networks and Applications, 19(2), 171– 209. https://doi.org/10.1007/s11036-013-0489-0 Cheng, H.-T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., … others. (2016). Wide & deep learning for recommender systems. Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, 7–10. https://doi.org/10.1145/2988450.2988454 Cho, Y. H., Kim, J. K., & Kim, S. H. (2002). A personalized recommender system based on web usage mining and decision tree induction. Expert Systems with Applications, 23(3), 329–342. https://doi.org/10.1016/S0957-4174(02)00052-0 Chorowski, J. K., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015). Attention-Based Models for Speech Recognition. Nips, 577–585. https://doi.org/10.1016/j.asr.2015.02.035 Christopher, M. (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. Retrieved from https://scholar.google.co.in/scholar?hl=en&as_sdt=0%2C5&q=C.+M.+Bishop%2C+Pattern+Reco gnition+and+Machine+Learning.+New+York%2C+NY%2C+USA%3A+Springer%2C+2006&btnG= Condli, M. K., Madigan, D., Lewis, D. D., & Posse, C. (1999). Bayesian Mixed-Effects Models for Recommender Systems. In ACM SIGIR ’99 Workshop on Recommender Systems: Algorithms and Evaluation. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=5B779C69031F4141C0276B05A0092 A38?doi=10.1.1.10.6090&rep=rep1&type=pdf Covington, P., Adams, J., & Sargin, E. (2016). Deep Neural Networks for YouTube Recommendations. Proceedings of the 10th ACM Conference on Recommender Systems - RecSys ’16, 191–198. https://doi.org/10.1145/2959100.2959190 Cui, P., Wang, X., Pei, J., & Zhu, W. (2017). A Survey on Network Embedding. Retrieved from 42 http://arxiv.org/abs/1711.08752 Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4), 303–314. https://doi.org/10.1007/BF02551274 Dalmia, A., & Gupta, M. (2018). Towards Interpretation of Node Embeddings. BigNet, (BigNet), 945– 952. https://doi.org/10.1145/3184558.3191523 Dawes, J., Meyer-Waarden, L., & Driesener, C. (2015). Has brand loyalty declined? A longitudinal analysis of repeat purchase behavior in the UK and the USA. Journal of Business Research, 68(2), 425–432. https://doi.org/10.1016/j.jbusres.2014.06.006 Dow, J. K., & Endersby, J. W. (2004). Multinomial probit and multinomial logit: a comparison of choice models for voting research. Electoral Studies, 23(1), 107–122. https://doi.org/10.1016/S02613794(03)00040-4 Ehrenberg, A., & Goodhardt, G. (2002). Double Jeopardy Revisited, Again. Marketing Research, 14(1), 40–42. https://doi.org/10.2307/1251818 Ehrenberg, A. S. C., Goodhardt, G. J., & Barwise, T. P. (1990). Double Jeopardy Revisited. Journal of Marketing, 54(3), 82. https://doi.org/10.2307/1251818 Ehrenberg, A. S. C., Uncles, M. D., & Goodhardt, G. J. (2004). Understanding brand performance measures: Using Dirichlet benchmarks. Journal of Business Research, 57(12 SPEC.ISS.), 1307– 1325. https://doi.org/10.1016/j.jbusres.2002.11.001 Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211. https://doi.org/10.1016/0364-0213(90)90002-E Erhan, D., Manzagol, P.-A., Bengio, Y., Bengio, S., & Vincent, P. (2009). The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training. In International Conference on Artificial Intelligence and Statistics (Vol. 5, pp. 153–160). Fader, P. S., & Hardie, B. G. S. (2008). Computing P(alive) Using the BG / NBD Model. Retrieved from http://brucehardie.com/notes/021/ Fader, P. S., Hardie, B. G. S., & Berger, P. D. (2004). Customer-Base Analysis with Discrete-Time Transaction Data. Fader, P. S., Hardie, B. G. S., & Lee, K. L. (2005). “Counting Your Customers” the Easy Way: An Alternative to the Pareto/NBD Model. Marketing Science, 24(2), 275–284. https://doi.org/10.1287/mksc.1040.0098 Fleder, D., & Hosanagar, K. (2009). Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. Management Science, 55(5), 697–712. https://doi.org/10.1287/mnsc.1080.0974 Fleder, D. M., & Hosanagar, K. (2007). Recommender systems and their impact on sales diversity. Proceedings of the 8th ACM Conference on Electronic Commerce - EC ’07, 192. https://doi.org/10.1145/1250910.1250939 Forman, G., & Scholz, M. (2009). Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement. HP Labs, 12(1), 49–57. https://doi.org/10.1145/1882471.1882479 Fortes, A. (2018). Public Datasets For Recommender Systems. Retrieved August 9, 2018, from https://github.com/caserec/Datasets-for-Recommneder-Systems 43 Fortuna, B., Fortuna, C., & Mladenić, D. (2010). Real-time news recommender system. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 6323 LNAI(PART 3), 583–586. https://doi.org/10.1007/978-3-642-15939-8_38 Frank Robert, H., & Cook Philip, J. (1995). The Winner-Take-All-Society. Penguin Books. Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:The+Winner-takeall+society#0%5Cnhttp://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:The+winn er-take-all+society+Why+the+Few+at+the+Top+Get+So+Much+More+Than+the+Rest+of+Us#0 Fujitsu. (2017). A Reference Model for High Performance Data Analytics(HPDA) using an HPC infrastructure. Retrieved from https://sp.ts.fujitsu.com/dmsp/Publications/public/wp-pf4hpchpda.pdf Gass, S. I., & Rapcsák, T. (2004). Singular value decomposition in AHP. European Journal of Operational Research, 154(3), 573–584. https://doi.org/10.1016/S0377-2217(02)00755-5 Gavrilov, M., Anguelov, D., Indyk, P., & Motwani, R. (2000). Mining The Stock Market: Which Measure Is Best? Acm Sigkdd, 487–496. https://doi.org/10.1145/347090.347189 Ge, M., Delgado-Battenfeld, C., & Jannach, D. (2010). Beyond accuracy: evaluating recommender systems by coverage and serendipity. … on Recommender Systems, 257–260. https://doi.org/10.1145/1864708.1864761 Gershman, A., & Meisels, A. (2010). A Decision Tree Based Recommender System. In G. Eichler, P. Kropf, U. Lechner, P. Meesad, & H. Unger (Eds.), International Conference on Innovative Internet Community Systems (I 2 CS) – Jubilee Edition 2010 (pp. 170–179). GI. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.1772&rep=rep1&type=pdf#pag e=171 Gilotte, A., Calauzènes, C., Nedelec, T., Abraham, A., & Dollé, S. (2018). Offline A/B testing for Recommender Systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (pp. 198–206). https://doi.org/10.1145/3159652.3159687 Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. AISTATS ’11: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, 15, 315– 323. https://doi.org/10.1.1.208.6449 Goel, S., Broder, A., Gabrilovich, E., & Pang, B. (2010). Anatomy of the Long Tail : Ordinary People with Extraordinary Tastes. Business, 48(5), 201–210. https://doi.org/10.1145/1718487.1718513 Goga, M., Kuyoro, S., & Goga, N. (2015). A Recommender for Improving the Student Academic Performance. Procedia - Social and Behavioral Sciences, 180(November 2014), 1481–1488. https://doi.org/10.1016/j.sbspro.2015.02.296 Goldberg, D., Nichols, D., Oki, B. M., & Terry, D. (1992). Using collaborative filtering to weave an information tapestry. Communications of the ACM, 35(12), 61–70. https://doi.org/10.1145/138859.138867 González-Briones, A., Rivas, A., Chamoso, P., Casado-Vara, R., & Corchado, J. M. (2018). Case-Based Reasoning and Agent Based Job Offer Recommender System. In Á. Herrero, B. Baruque, J. Sedano, H. Quintián, & E. Corchado (Eds.), International Joint Conference SOCO’18-CISIS’18-ICEUTE’18 (Vol. 369, pp. 21–33). Cham: Springer International Publishing. https://doi.org/10.1007/978-3319-94120-2_3 Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Retrieved from 44 http://www.deeplearningbook.org Goodhardt, G. J., Ehrenberg, A. S. C., & Chatfield, C. (1984). The Dirichlet : A Comprehensive Model of Buying Behaviour. Journal of the Royal Statistical Society, 147(5), 621–655. https://doi.org/10.2307/2981696 Google. (2018a). Play games with a Gamer ID. Retrieved July 29, 2018, from https://support.google.com/googleplay/answer/2954594?hl=en Google. (2018b). Use Google Play Family Library. Retrieved July 29, 2018, from https://support.google.com/googleplay/answer/7007852?hl=en Graham, C., Bennett, D., Franke, K., Henfrey, C. L., & Nagy-Hamada, M. (2017). Double Jeopardy – 50 years on. Reviving a forgotten tool that still predicts brand loyalty. Australasian Marketing Journal, 25(4), 278–287. https://doi.org/10.1016/j.ausmj.2017.10.009 Graves, A., Wayne, G., & Danihelka, I. (2014). Neural Turing Machines, 1–26. https://doi.org/10.3389/neuro.12.006.2007 Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., … Hassabis, D. (2016). Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626), 471–476. https://doi.org/10.1038/nature20101 Greenacre, M. J. (1988). Clustering the rows and columns of a contingency table. Journal of Classification, 5(1), 39–51. https://doi.org/10.1007/BF01901670 Gupta, S., Hanssens, D., Hardie, B., Kahn, W., Kumar, V., Lin, N., … Sriram, S. (2006). Modeling customer lifetime value. Journal of Service Research, 9(2), 139–155. https://doi.org/10.1177/1094670506293810 Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., … Wang, X. (2018). Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective. Proceedings - International Symposium on High-Performance Computer Architecture, 2018–Febru, 620–629. https://doi.org/10.1109/HPCA.2018.00059 Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1), 5–53. https://doi.org/10.1145/963770.963772 Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786), 504–507. https://doi.org/10.1126/science.1127647 Hochreiter, S., & Urgen Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2), 251–257. https://doi.org/10.1016/0893-6080(91)90009-T Hsu, C. W., & Lin, C. J. (2002). A comparison of methods for multiclass support vector machines. IEEE Transactions on Neural Networks, 13(2), 415–425. https://doi.org/10.1109/72.991427 Huang, C.-Y. (2011). Excess Loyalty in Online Retailing. International Journal of Electronic Commerce, 16(2), 115–134. https://doi.org/10.2753/JEC1086-4415160206 Huggett, M., Hoos, H., & Rensink, R. (2007). Cognitive principles for information management: The Principles of Mnemonic Associative Knowledge (P-MAK). Minds and Machines, 17(4), 445–485. 45 https://doi.org/10.1007/s11023-007-9080-4 Hunter, T. (2016). Deep Learning with Apache Spark and TensorFlow. Retrieved from https://databricks.com/blog/2016/01/25/deep-learning-with-apache-spark-andtensorflow.html Iwata, T., Saito, K., & Yamada, T. (2008). Recommendation Method for Improving Customer Lifetime Value. IEEE Transactions on Knowledge and Data Engineering, 20(9), 1254–1263. https://doi.org/10.1109/TKDE.2008.55 Jahrer, M., Töscher, A., & Legenstein, R. (2010). Combining predictions for accurate recommender systems. Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’10, 693. https://doi.org/10.1145/1835804.1835893 Jeuland, A. P., Bass, F. M., & Wright, G. P. (1980). A Multibrand Stochastic Model Compounding Heterogeneous Erlang Timing and Multinomial Choice Processes. Operations Research, 28(2), 255–277. https://doi.org/10.1287/opre.28.2.255 Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., … Darrell, T. (2014). Caffe: Convolutional Architecture for Fast Feature Embedding. https://doi.org/10.1145/2647868.2654889 Jin, R., & Si, L. (2004). A bayesian approach toward active learning for collaborative filtering. Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, 278–285. Retrieved from http://dl.acm.org/citation.cfm?id=1036877 Ju, J. Y., Choi, I. Y., Kim, J. K., & Moon, H. S. (2017). Reinforcement Learning for Profit Maximization of Recommender Systems Reinforcement Learning for Profit Maximization of Recommender Systems Completed Research Paper. In Proceedings of the P re-ICIS 2017 S IGDSA Symposium. Seoul, South Korea. Retrieved from https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1003&context=sigdsa2017 Karney, C. F. F. (2013). Algorithms for geodesics. Journal of Geodesy, 87(1), 43–55. https://doi.org/10.1007/s00190-012-0578-z Karsmakers, P., Pelckmans, K., & Suykens, J. A. K. (2007). Multi-class kernel logistic regression : a fixedsize implementation. In Advances in Neural Information Processing Systems (Vol. 20, pp. 1177– 1184). Katsov, I. (2018). Introduction to Algorithmic Marketing - Artificial Intelligence for Marketing Operations. Grid Dynamics. Kim, H., Park, J., Jang, J., & Yoon, S. (2016). DeepSpark: A Spark-Based Distributed Deep Learning Framework for Commodity Clusters, (Nips). Retrieved from http://arxiv.org/abs/1602.08191 Kim, K. (2011). Customer Need Type Classification Model using Data Mining Techniques for Recommender Systems, 5(8), 279–284. Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. In ICLR 2015 (pp. 1–15). https://doi.org/http://doi.acm.org.ezproxy.lib.ucf.edu/10.1145/1830483.1830503 Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., & Newell, C. (2012). Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction, 22(4–5), 441–504. https://doi.org/10.1007/s11257-011-9118-4 46 Kontschieder, P., Fiterau, M., Criminisi, A., & Bulo, S. R. (2015). Deep Neural Decision Forests. In 2015 IEEE International Conference on Computer Vision (ICCV) (pp. 1467–1475). IEEE. https://doi.org/10.1109/ICCV.2015.172 Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix Factorization Techniques for Recommender Systems. Computer, 42(8), 30–37. https://doi.org/10.1109/MC.2009.263 Koutrika, G. (2018). Modern Recommender Systems: from Computing Matrices to Thinking with Neurons. https://doi.org/10.1145/3183713.3197389 Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances In Neural Information Processing Systems, 1–9. https://doi.org/http://dx.doi.org/10.1016/j.protcy.2014.09.007 Landset, S., Khoshgoftaar, T. M., Richter, A. N., & Hasanin, T. (2015). A survey of open source tools for machine learning with big data in the Hadoop ecosystem. Journal of Big Data, 2(1), 24. https://doi.org/10.1186/s40537-015-0032-1 Le, Q. V., & Mikolov, T. (2014). Distributed Representations of Sentences and Documents, 32. https://doi.org/10.1145/2740908.2742760 LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539 Lehal, G. S. (2007). Design and Implementation of Punjabi Spell Checker. International Journal of Systemics, Cybernetics and Informatics, 70–75. https://doi.org/10.17485/ijst/2015/v8i27/83917 Leshno, M., Lin, V. Y., Pinkus, A., & Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6), 861– 867. https://doi.org/10.1016/S0893-6080(05)80131-5 Li, Y. M., Wu, C. Te, & Lai, C. Y. (2013). A social recommender mechanism for e-commerce: Combining similarity, trust, and relationship. Decision Support Systems, 55(3), 740–752. https://doi.org/10.1016/j.dss.2013.02.009 Liu, C., & Zhou, W. (2010). An improved HeatS+ ProbS hybrid recommendation algorithm based on heterogeneous initial resource configurations. ArXiv Preprint ArXiv:1005.3124, 1–6. https://doi.org/10.1016/j.physa.2012.06.034 Liu, X., Aggarwal, C., Li, Y.-F., Kong, X., Sun, X., & Sathe, S. (2016). Kernelized Matrix Factorization for Collaborative Filtering. Proceedings of the 2016 SIAM International Conference on Data Mining, 378–386. https://doi.org/10.1137/1.9781611974348.43 Lü, L., Medo, M., Yeung, C. H., Zhang, Y.-C., Zhang, Z.-K., & Zhou, T. (2012). Recommender systems. Physics Reports, 519(1), 1–49. https://doi.org/10.1016/j.physrep.2012.02.006 Luaces, O., Díez, J., Barranquero, J., del Coz, J. J., & Bahamonde, A. (2012). Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence, 1(4), 303–313. https://doi.org/10.1007/s13748-012-0030-x Maillo, J., Ramírez, S., Triguero, I., & Herrera, F. (2017). kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data. Knowledge-Based Systems, 117, 3–15. https://doi.org/10.1016/j.knosys.2016.06.012 Maroco, J., Silva, D., Rodrigues, A., Guerreiro, M., Santana, I., & De Mendonça, A. (2011). Data mining 47 methods in the prediction of Dementia: A real-data comparison of the accuracy, sensitivity and specificity of linear discriminant analysis, logistic regression, neural networks, support vector machines, classification trees and random forests. BMC Research Notes, 4. https://doi.org/10.1186/1756-0500-4-299 Michmizos, D., Koutsouraki, E., Asprodini, E., & Baloyannis, S. (2011). Synaptic Plasticity: A Unifying Model to Address Some Persisting Questions. International Journal of Neuroscience, 121(6), 289– 304. https://doi.org/10.3109/00207454.2011.556283 Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. Retrieved from http://arxiv.org/abs/1301.3781 Mild, A., & Natter, M. (2002). Collaborative filtering or regression models for Internet recommendation systems? Journal of Targeting, Measurement and Analysis for Marketing, 10(4), 304–313. https://doi.org/10.1057/palgrave.jt.5740055 Miyahara, K., & Pazzani, M. J. (2000). Collaborative Filtering with the Simple Bayesian Classifier. In PRICAI 2000 Topics in Artificial Intelligence (Vol. 8046, pp. 679–689). Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44533-1_68 Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014). On the Number of Linear Regions of Deep Neural Networks. https://doi.org/10.1007/978-1-4471-5779-3_4 Moritz, P., Nishihara, R., Stoica, I., & Jordan, M. I. (2015). SparkNet: Training Deep Networks in Spark, 1–12. Retrieved from http://arxiv.org/abs/1511.06051 Neumann, J. von. (1945). First Draft of a Report on the EDVAC. Nickolls, J., & Dally, W. J. (2010). The GPU Computing Era. IEEE Micro, 30(2), 56–69. https://doi.org/10.1109/MM.2010.41 O’Leary, D. E. (2014). Embedding AI and crowdsourcing in the big data lake. IEEE Intelligent Systems, 29(5), 70–73. https://doi.org/10.1109/MIS.2014.82 O’Mahony, M. P., Cunningham, P., & Smyth, B. (2010). An Assessment of Machine Learning Techniques for Review Recommendation. In Artificial Intelligence and Cognitive Science : 20th Irish Conference (pp. 241–250). https://doi.org/10.1007/978-3-642-17080-5_26 Oku, K., Nakajima, S., Miyazaki, J., & Uemura, S. (2006). Context-aware SVM for context-dependent information recommendation. Proceedings - IEEE International Conference on Mobile Data Management, 2006, 5–8. https://doi.org/10.1109/MDM.2006.56 Olah, C., & Carter, S. (2016). Attention and Augmented Recurrent Neural Networks. Retrieved August 12, 2018, from https://distill.pub/2016/augmented-rnns/ Osogami, T. (2017). Boltzmann machines and energy-based models. Retrieved from http://arxiv.org/abs/1708.06008 Ostuni, V. C., Di Noia, T., Mirizzi, R., & Di Sciascio, E. (2014). Top-N recommendations from implicit feedback leveraging linked open data. CEUR Workshop Proceedings, 1127, 20–27. https://doi.org/10.1145/2507157.2507172 Oussous, A., Benjelloun, F. Z., Ait Lahcen, A., & Belfkih, S. (2017). Big Data technologies: A survey. Journal of King Saud University - Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2017.06.001 54 𝜏𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑𝑠𝑗𝑇=∑∑𝑠𝑑𝑗𝑡 𝑇 𝑡=1𝑑∈𝒟𝑗 Definition 8. Market Share. Let 𝑀𝑆𝑘 denote the market share of the mobile app store 𝑘∈𝒢 such that, 𝑀𝑆𝑘=∑𝑠𝑖𝑘𝑖∈ℳ ∑ ∑ 𝑠𝑖𝑗𝑖∈ℳ𝑗∈𝒢 Definition 9. User Churn. Let 𝑝𝑖𝑘 be the individual churn probability in mobile app store 𝑘∈𝒢 for user 𝑖∈ℳ conditional on the number of app downloads in all mobile app store 𝑗∈𝒢 in the period such that, 𝑝𝑖𝑘=𝑝(𝑠𝑖𝑘>0|𝜏𝑠𝑖𝑗) Definition 10. Mobile App Store Churn. Let 𝑝𝑗 be the churn rate of mobile app store 𝑗∈𝒢 such that, 𝑝𝑗=∑𝑝𝑖𝑗𝑖∈𝒟 𝐷 Definition 11. Category App Download Frequency. Let 𝑛 denote the app download frequency of mobile app store category 𝒢 such that, 𝑛=∑ ∑ 𝑠𝑖𝑗𝑗∈𝒢𝑖∈ℳ|ℳ| =∑𝜏𝑠𝑖𝑗 𝑖∈ℳ |ℳ| Definition 12. Customer Lifetime Value (𝐶𝐿𝑉) until 𝑇. Let 𝐶𝐿𝑉𝑖𝑇 denote the Customer Lifetime Value of user 𝑖∈𝒟𝑗 until period 𝑇 such that (Blattberg et al., 2008, pp. 108–109), 𝐶𝐿𝑉𝑖𝑇=∑(1−𝑝𝑗)(𝑅𝑖𝑗𝑡−𝐶𝑖𝑗𝑡) (1+𝛿)𝑡−1 𝑇 𝑡=1 Where 𝑅𝑖𝑗𝑡 is the revenue generated by user 𝑖 in period 𝑡, 𝐶𝑖𝑡 is the cost of serving the user 𝑖 on moment 𝑡 and 𝛿 is the discount rate. 55 Definition 13. Total Customer Lifetime Value until 𝑇. Let 𝜏𝐶𝐿𝑉𝑗𝑇 denote the total Customer Lifetime Value of the mobile app store 𝑗∈ℳ until period 𝑇. 𝜏𝐶𝐿𝑉𝑗𝑇 =∑𝐶𝐿𝑉𝑑𝑇 𝑑∈𝒟𝑗 Definition 14. App Download Probability. Let 𝑙𝑗 denote the probability of a download occurring on mobile app store 𝑗∈ℳ under a frequentist assumption such that, 𝑙𝑗=lim 𝑇→∞ 1 𝜏𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑𝑠𝑗𝑇 Definition 15. App Download Value. Let 𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑 denote the value of an app download for mobile app store 𝑗∈ℳ, such that, 𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑=lim 𝑇→∞ 𝜏𝐶𝐿𝑉𝑗𝑇 𝜏𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑𝑠𝑗𝑇=lim 𝑇→∞(𝜏𝐶𝐿𝑉𝑗𝑇)×𝑙𝑗 We will now present our main assumptions under the form of axioms that are required to define our statistical estimators. These assumptions are the basic assumptions behind the NBD-Dirichlet model (Goodhardt et al., 1984) and are grounded in the empirical generalizations that have been observed over several decades across markets and geographies (Graham, Bennett, Franke, Henfrey, & NagyHamada, 2017). In particular these patterns have also been observed in the mobile app economy (Zhong & Michahelles, 2013) and amongst online retailers (Huang, 2011). Therefore, we believe these to be a set of realistic assumptions. Axiom 1. The probability of a user doing 𝑛 downloads in a given period follows a Poisson Distribution. 𝑝(𝑠𝑖𝑗𝑡=𝑘) ~ 𝑒−𝜇𝑖𝑑𝑎𝑦𝑠×𝜇𝑖𝑘 𝑘! Axiom 2. Individual user app download frequencies vary according to a Gamma Distribution. 𝑠𝑖 ~ e−𝜇𝑖𝐾 𝑀𝜇𝑖𝐾−1 Γ(𝐾)(𝑀 𝐾)𝐾 56 Axiom 3. User choice of mobile app store over 𝑘 successive choices follows a Multinomial Distribution. 𝑐𝑖𝑗 ~ 𝑘! ∏𝑠𝑖𝑗𝑡 𝑇𝑡=1 ∏ 𝛼𝑔 𝑔∈𝒢 Axiom 4. Individual user mobile app store choice frequencies vary according to a Dirichlet Distribution. 𝜃𝑖𝑔 ~ ∏(∑𝑠𝑖𝑔𝑡 𝑇𝑡=1 )𝛼𝑔−1Γ(∑𝛼𝑔𝑔∈𝒢 ) 𝑔∈𝒢 ∏Γ(𝛼𝑔) 𝑔∈𝒢 Axiom 5. The mobile app store choice probabilities and the average app download frequencies of the different users 𝑚∈ℳ are distributed independently. Definition 15. NBD-Dirichlet Model (Conditional App Downloads Estimator). The set of axioms imply that we can obtain the probability 𝑝(𝑟𝑗|𝑛) of a user 𝑖∈ℳ doing 𝑟𝑗∈ℕ app downloads on period 𝑡 in mobile app store 𝑗∈𝒢 amongst 𝑛𝑖 app downloads by employing a compound distribution such that (Goodhardt et al., 1984), [𝑘! ∏𝑠𝑖𝑗𝑡 𝑇𝑡=1 ∏ 𝛼𝑗 𝑗∈𝒢 ∧ 𝛼𝑗 ∏(∑𝑠𝑖𝑗𝑡 𝑇𝑡=1 )𝛼𝑗−1Γ(∑𝛼𝑗𝑗∈𝒢 ) 𝑗∈𝒢 ∏Γ(𝛼𝑗) 𝑗∈𝒢 ] ∧𝑛[𝑒−𝜇𝑖𝑑𝑎𝑦𝑠×𝜇𝑖𝑘 𝑘! ∧ 𝜇𝑖e−𝜇𝑖𝐾 𝑀𝜇𝑖𝐾−1 Γ(𝐾)(𝑀 𝐾)𝐾 ] Where 𝛼𝑗 is the app download propensity of mobile app store 𝑗, 𝑀 and 𝐾 are model parameters and 𝑑𝑎𝑦𝑠 refers to the length (in days) of each time period 𝑡. Therefore, 𝑝(𝑟𝑗|𝑘)~(𝑘 𝑟𝑗)Β(𝛼𝑗+𝑟𝑗,𝑆−𝛼𝑗+𝑘−𝑟𝑗) Β(α𝑗,𝑆−α𝑗) Where 𝑆 is the diversity of usage behavior in the category (𝑆=∑α𝑗𝑗 ) (Bound, 2009). We can assume that 𝑀𝑆𝑗=𝛼𝑗/𝑆 (Wright et al., 2002). Remark 1. Mobile App Store Churn Rate Estimator. From Definition 15 and the set of axioms it follows that we can obtain an estimator 𝑝𝑗 for 𝑝𝑗 such that, 𝑝𝑗=𝑝(𝑟𝑗=0|𝐺) Remark 2. App Download Probability Estimator. From Definition 15 and the set of axioms it follows that we can obtain an estimator 𝑙󰆹𝑗 for 𝑙𝑗 such that, 𝑙󰆹𝑗=𝑝(𝑟𝑗=1|𝐺) 57 Definition 16. Average Mobile App Store Customer Lifetime Value Estimator. Let 𝐶𝐿𝑉 𝑗𝑇be an estimator for 𝐶𝐿𝑉𝑗𝑇 such that, 𝐶𝐿𝑉 𝑗𝑇=∑(1−𝑝𝑗)(𝑅𝑡−𝐶𝑡) (1+𝛿)𝑡−1 𝑇 𝑡=1 Definition 17. Total Customer Lifetime Value Estimator over finite horizon ℎ. Let 𝜏𝐶𝐿𝑉𝑗ℎ be an estimator for 𝜏𝐶𝐿𝑉𝑗ℎ such that, 𝜏𝐶𝐿𝑉𝑗ℎ =lim 𝑇→ℎ𝐷𝐶𝐿𝑉 𝑗𝑇 Definition 17. App Download Value Estimator. Let 𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑 be an estimator for 𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑 such that, 𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑=𝜏𝐶𝐿𝑉𝑗ℎ×𝑙󰆹𝑗= =𝑝(𝑟𝑗=1|𝑛)𝐷lim 𝑇→ℎ𝐶𝐿𝑉 𝑗𝑇= =𝑝(𝑟𝑗=1|𝑛)𝐷lim 𝑇→ℎ∑(1−𝑝𝑗)(𝑅𝑡−𝐶𝑡) (1+𝛿)𝑡−1 𝑇 𝑡=1 = =∑𝐷(1−𝑝𝑗)(𝑅𝑡−𝐶𝑡) 𝑝(𝑟𝑗=1|𝑛) (1+𝛿)𝑡−1 ℎ 𝑡=1 Replacing 𝑝𝑗 and 𝑝(𝑟𝑗=1|𝑛) with their respective expressions we get, 𝑉𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑=∑𝑛𝐷(𝑅𝑡−𝐶𝑡)[Β(α𝑗,𝑆−α𝑗)−Β(𝛼𝑗,𝑆−𝛼𝑗+𝑛)]Β(𝛼𝑗+1,𝑆−𝛼𝑗+𝑛−1) (1+𝛿)𝑡−1Β(α𝑗,𝑆−α𝑗)2 ℎ 𝑡=1