The effect of Price Promotions on Online User Reviews

THE EFFECTS OF PRICE PROMOTIONS ON ONLINE USER REVIEWS by Bruno Miguel Saldanha Sista Dissertation for the Master in Marketing Oriented by: Prof. Dr. Beatriz da Graça Luz Casais Prof. Dr. Nuno Alexandre Meneses Bastos Moutinho 2015 Biographical Sketch Bruno Miguel Saldanha Sista was born in Vila Nova de Gaia, Portugal, on the 28th June 1991. Showing a love for letters and a head for numbers from a very early age, he was admitted to elementary school at 5 years old and was consistently an honour roll student throughout primary, middle and high school. At 14 he chose to attend the Socioeconomic Sciences course, driven by his increasing interest in business and finance, a trait inherited from his entrepreneur mother, manager of a local computer retailing business, where Bruno spent most of his afternoons nurturing a deep passion for technology. Graduating high school as top of his class in 2008, he went on to attend the Bachelor in Economics at the University of Porto – Faculdade de Economia. There, he was able to further develop his understanding of business and trade markets, always complementing his classroom learning with the soft-skill development opportunities that only students’ organizations can provide (as coordinator of the Marketing department at NEV/EXUP as well as team leader at eCOROmia – Choir of the Faculty of Economics of Porto). Bruno also participated in the ERASMUS exchange programme, through which he attended courses in the Master of Business Administration at the Universitá degli Studi di Roma Tor Vergata for a semester in 2011. He finally graduated on the 12th December 2012. In April 2013 he was recruited by Sage Portugal, a subsidiary of Sage Group plc, to fill the role of Controller in the Financial Planning & Analysis department. His passion for numbers and complex analytical puzzles and his technological savvy have been useful since then, in his crusade for timely KPI reports and comprehensive dashboards. In September of the same year he embraced the challenge of pursuing further education, and finally enrolled in the Master in Marketing at his alma mater, FEP-UP. Acknowledgments My deepest gratitude goes to my advisor, Prof. Beatriz Casais, for her caring support of my research project, for encouraging me to be more inquisitive, and for motivating me to be more ambitious every step of the way. Prof. Nuno Moutinho, my co-advisor, takes credit for helping me focus on a research proposition that was both purposeful and impactful. His insightful comments and expertise were of great help throughout all stages of my study. As I look back to the two years that have passed since I attended the first class of this course, I can’t go without congratulating the academic staff of the Master’s degree in Marketing for the incredible learning experience they provided, as well as my colleagues for the companionship and support we shared as we approach, together, the conclusion of this chapter of our academic lives. My whole team at Sage (Filipa, Patrícia, Ana and Isabel) deserves an honourable mention for their infinite comprehension and patience for my occasional “sleep deprivation-induced” blunders at work and my more-than-occasional tardiness, when my late-night research efforts called for it. It was in no small part thanks to your guidance and the experience you offered me that I was able to tackle this incredible challenge. Last, but certainly not least, my heartfelt thanks go to my family and friends for their continuous support and enthusiasm, for standing by me through the good times and bad, for helping me stay sane and confident in myself, and for generally making my world a sunnier place. Abstract The aim of this research is to analyse how pricing strategies (specifically through promotional discounts) can lead to fluctuations in user review scores of products in an online marketplace. This will hopefully shed some light on the role of price in the customers’ pre-purchase expectations, and the post-purchase evaluation of their consumption experience. To achieve that goal, we’ve done extensive research into the concepts of online communities, electronic word of mouth, and customer satisfaction, in an attempt to understand the mechanisms behind fluctuations in online recommendations after the occurrence of promotional discounts. A practical application of those concepts was made through observation of consumer behaviour in users of the Steam platform, an online software distribution service, collecting review scores over the course of 2 months. Applying change point analysis methods, we confirm that promotional discounts not only had a significant effect on the volume of reviews posted, but also caused a fluctuation in the products’ user ratings, disrupting the otherwise stable process of word of mouth generation, causing variations in review scores that can be either positive or negative. Finally, we reaffirm the importance of future research on the subjects of electronic word of mouth and online recommendation systems, proposing the addition of further variables to this analysis, including product attributes and discount rates, which will allow for more definite answers on how pricing strategies, when paired with adequate online feedback management policies, can be used to generate more business in online markets. Keywords: Online recommendations, Electronic Word of Mouth, Virtual Communities, User Reviews, Product ratings, Customer satisfaction Resumo O objectivo desta investigação foi o de analisar como alterações na estratégia de pricing de um produto (especificamente, promoções de preços) podem causar flutuações nas recomendações de utilizadores em plataformas de comércio online. A finalidade é trazer uma nova luz sobre o papel do preço na formação de expectativas pré-compra por parte dos consumidores, e as avaliações pós-compra das suas experiências de consumo. Para responder a esse desafio, foi feita uma extensiva revisão bibliográfica sobre os conceitos de comunidades virtuais, electronic word of mouth e satisfação do consumidor, numa tentativa de compreender os mecanismos que estão por detrás das flutuações nos online review scores depois da ocorrência de descontos de preço. Foi ainda feita uma aplicação prática destes conceitos através da observação do comportamento dos utilizadores da plataforma Steam, um serviço de distribuição digital de videojogos, tendo sido recolhidos dados de review scores ao longo de 2 meses. Aplicando métodos de change point analysis (Análise de pontos de mudança), confirmouse que descontos promocionais têm não só um efeito significativo no volume de recomendações publicadas online, mas também na apreciação média dos consumidores, criando uma disrupção no processo de geração de word of mouth sob a forma de variações nos review scores que podem ser positivas ou negativas. Finalmente, reafirmamos a necessidade de investigação futura sobre os temas de electronic word of mouth e sistemas de recomendações online, propondo a adição de mais variáveis para esta análise, incluindo atributos ao nível do produto e as taxas de desconto, que permitirão respostas mais completas e com nível de confiança superior para estas e outras questões de investigação. A finalidade será investigar a viabilidade de utilizar estratégias de pricing, combinadas com políticas eficazes de gestão de feedback online, para gerar maior volume de vendas nos mercados online. Palavras-chave: Recomendações Online, Electronic Word of Mouth, Comunidades Virtuais, User Reviews, Product ratings, Satisfação do Consumidor. Table of Contents I. Introduction .................................................................................................. 2 II. Literature Review ..................................................................................... 4 i. Communities and their social value ............................................................ 4 ii. Word of mouth ........................................................................................ 5 iii. Motivations for Word of Mouth engagement ......................................... 7 iv. Online communities ................................................................................ 8 v. Online Marketplaces and Online User Reviews ................................... 10 vi. Implications of Online Recommendations and Review Scores ............ 12 vii. Customer Satisfaction theories .............................................................. 13 viii. Systems for Online User Reviews ..................................................... 15 III. Research proposal and Methodology .................................................... 19 i. Research object ......................................................................................... 20 ii. Data collection ...................................................................................... 22 iii. Sample Selection and Data Treatment .................................................. 24 iv. Main variables for research ................................................................... 25 v. Change Point Analysis and CUSUM charts .......................................... 27 IV. Results and Findings ............................................................................... 29 i. New reviews published daily by product .................................................. 29 ii. Average review scores by product ........................................................ 30 V. Discussion ................................................................................................ 35 VI. Conclusion ............................................................................................... 39 i. Conclusions ............................................................................................... 39 ii. Limitations and future research ............................................................. 40 VII. References ............................................................................................ 42 VIII. Appendices ........................................................................................... 49 Appendix I – Autocorrelation Factors for Number of Reviews ...................... 49 Appendix II – Change Point Detection test for New Reviews ........................ 52 Appendix III – Change Point Detection test for Review Scores ..................... 54 List of Tables Table 1 - Example of data extracted from the Steam Storefront ........................ 22 Table 2 - Distributional statistics for Product Review Scores ........................... 31 List of Figures Figure 1 - Typology of Virtual Communities proposed by Porter (2004) ........... 9 Figure 2 - Example of Amazon product review summary ................................. 15 Figure 3 - Example of booking.com review system .......................................... 15 Figure 4 - ebay.com seller panel on a product page (example) ......................... 16 Figure 5 - ebay.com page for the same seller as in Figure 4 ............................. 16 Figure 6 - Example of Steam Store Product Page, with the Product Reviews information highlighted .................................................................................................. 17 Figure 7 - Example of Customer Reviews section in a Steam product page ..... 17 Figure 8 - Screenshot from the Steam Storefront product Search, illustrating the main attributes of a product, including product reviews and discounts .......................... 22 Figure 9 - Reviews Published Daily by Product ................................................ 29 Figure 10 – Time Series plot for Review Scores ............................................... 30 Figure 11 - Time Series Boxplot for Product Review Scores ............................ 31 Figure 12 - Standardized Review Scores plot .................................................... 32 Figure 13 - CUSUM Chart for Review Scores (Means) .................................... 33 Figure 15 – Illustration of Supply and Demand in a Posted-Price market ........ 35 Figure 16 - Negative product review published after a price reduction ............ 37 2 I. Introduction The advent of the commercial Internet allowed for the development of new communication tools and several new forms of interaction between users online. The Web 2.0, in turn, gave its users new powers to contribute with content to the network and bringing a social component to most online experiences (Solomon & Schrum, 2007). Several services have appeared on the web appealing to the “participatory culture”, from blogs to social networks, inviting the user to take part in the creation of social, economic and cultural value (Jöckel et al., 2008). e-Commerce is perhaps one of the most impactful new paradigms to emerge from this trend. The appearance of dotcoms such as Amazon.com and ebay.com, pioneers in the creation of online marketplaces that facilitate the fulfilment of transactions through the web, has come to revolutionize the way citizens of the Internet (individuals or companies) relate to trade and shopping. These marketplaces are dependent on the participation of consumers, their main selling point being the direct interaction between sellers, prospects and buyers, setting them apart from physical points of sale (Pavlou & Gefen, 2004). Social Commerce is a relatively recent concept, appearing under the umbrella of the Web 2.0 (Hajli, 2015). The modern user is no longer limited to the information producers/retailers choose to make available for their merchandise and services, or articles published in the specialized press, instead sourcing his information to his peers, who share first-hand opinions and reviews of their experiences with those products (Schafer et al., 1999). The creation of this collective intelligence has brought such impact on the decision-making processes for online shopping, that Electronic Word of Mouth (henceforth, eWOM) and User-Generated Content (UGC) have become two of the most discussed subjects between marketers (See-To et al., 2014), who now more than ever recognize the effects of customers participating in the creation of brand equity, either by increasing notoriety and reach of the brand, or through the part they play in changing other users’ perception of quality of the products. Customer feedback management and monitoring of user reviews in online platforms must be a concern addressed in any marketing plan, since these public manifestations of word of mouth can have very a positive effect on the generation of leads and business, but also a notably negative impact in the event negative buzz starts to circulate online (Chen & Xie, 2008). 9 Figure 1 - Typology of Virtual Communities proposed by Porter (2004) Regardless of types, Porter goes on to list 4 common attributes of all virtual communities:  Purpose – the common goal or interest shared by the community members (Jones & Rafaeli, 2000)  Place – not necessarily a physical space, so much as a sense of co-presence of members of a community and which can be metaphorically attributed to the “website” or “electronic address” (Harrison & Dourish, 1996)  Platform – defined by the technological mediums and infrastructures that allow individuals to communicate within a community.  Population – the social structure and pattern of interaction between members – communities can be small groups or networks, and the ties between members can vary in strength and nature. Another concept highly relevant in this study is that of virtual publics, “computermediated spaces, whose existence is relatively transparent and open, that allow groups of individuals to attend and contribute to a similar set of computer-mediated interpersonal interactions” (Jones & Rafaeli, 2000). While in the earlier stages of adoption these technologies were reserved to demographics who had enough disposable income to afford the cost of entry – resulting in a network of individuals homogeneous in economic standing and education (Barlow, 1995) – as the equipment and services become more and more accessible, the number of Internet groups and the diversity of virtual publics have increased exponentially. Online communities have been a hot topic in recent marketing and management publications, as companies focus their research on how to effectively harness the power of online communities and social media (Nambisan & Watt, 2011;Weinberg et al., 2013). 10 v. Online marketplaces and online user reviews We have already discussed the evolution of the Internet as a revolutionary channel of communication that paved the way to discussions unchained by geographic restrictions or the traditional social network paradigm. Initially, the Internet allowed the circulation of word of mouth through 1 to 1 media (e-mail messages) or 1 to many (via mailing lists and later through the proliferation of personal webspaces). But the revolution of Web 2.0 was game-changing in the way individuals interact online, allowing the development of tools for many-to-many communication, either through the modern social networks, forums or notice boards, and the broadcasting of user-generated content through social platforms such as blogs and YouTube or similar media-streaming websites (Solomon & Schrum, 2007). Product-related content can, in turn, be publically published through specialized platforms like epinions.com, or through online marketplaces (Lee & Youn, 2009). e-Commerce has been a part of the Internet for as long as it has been open to forprofit organizations, who immediately started pursuing opportunities to generate business leads via the new technology. Today, with an Internet penetration over 40% worldwide and growing (internetlivestats.com, June 2015), and PwC’s Global Total Retail Consumer Survey (PwC, February 2015) reporting over 50% of their respondents shop Online at least on a monthly basis, e-Commerce has proven to be essential for any business to reach out to their full market potential. But the backbone of e-Commerce is no longer exclusively comprised of companies reaching out to potential customers through their proprietary websites. The new reality of the Internet is oriented towards the creation of third-party platforms that serve as hubs for supply and demand, meeting points for buyers and sellers, who are provided with infrastructure and tools to communicate with each other and close deals (Pavlou & Gefen, 2004). Such is the case of ebay.com and Amazon, two of the largest online marketplaces today. In these platforms, sellers can post their products, including more or less detailed descriptions, specifications, prices and even promotions. Buyers can contact the seller, engage in negotiation and close deals, and provide feedback on several aspects of the purchase/consumption experience, such as the quality of products, price, the quality of the interaction with the seller, and the efficiency of delivery (Schafer et al., 1999). 11 Online marketplaces have recognized the potential of having recommendations systems and public review systems, where consumer opinions are gathered and often condensed in the form of review scores, or ratings, giving the customer a greater range of information about the products on sale without having to visit other websites (Chen & Xie, 2008). Besides users perceiving value in verbalizing their opinions online for the same motives we’ve discussed earlier, Hennig-Thurau (2004) also studied why users read those opinions. The most common goals are to “save decision making-time and make better decisions”, but there are other factors such as remuneration or the sense of belonging to a community, that drive users to consume word of mouth. Furthermore, the fact that communication in online platforms is durable and information is stored and visible for most visitors to read, brings a new light to the “community involvement” motivation for engaging in eWoM. Here, more than in traditional discussions, that contribution is not completely altruistic. By sharing their opinions through the Internet, users of web-based opinion platforms hope to motivate others to do the same, increasing the flux of information. That way, for example, if they encounter some problem with their product, they’re more likely to find someone who has encountered the same issue before and knows how to solve it. And, in turn, that is more likely to happen once the platform has reached a critical mass of users and interactions (Peddibhotla & Subramani, 2007). Users who share their opinion in search of recognition, self-enhancement or social benefit, can also see the Internet as a facilitator. Online marketplaces often recognize a user’s frequency of activity through reputation systems, letting shoppers access other users’ review history, which ultimately allows the perception of expertise and how helpful they have been in the past (Hu et al., 2008). Shen et al. (2012) even studied the competition for attention between online reviewers and their strategic perspective on content creation. Several other, more recent, studies have confirmed that online reviews have an influence on the process of information adopters by future buyers (Cheung, 2014), going as far as arguing that herd behaviour can lead shoppers to bias their opinions on others’ compelling arguments and credibility, leading to a tendency for imitation on the generation of Word of Mouth (Shen et al., 2014). 12 vi. Implications of online recommendations and review scores The idea that online reviews influence the consumer’s decision making process has been proven true by several studies. Senecal and Nantel (2004) conducted an online experiment that presented subjects with several combinations of brand websites, usergenerated recommendation sources, and products, reaching the conclusion that online recommendations acted as reliable information sources in the pre-purchase information seeking phase. As a consequence, “subjects who consulted product recommendations selected recommended products twice as often as subjects who did not consult recommendations.” (Senecal & Nantel, 2004, p.1) Based on those and similar results, online word of mouth has been suggested by several authors as a promising variable in forecasting future sales, particularly in the film industry (Dellarocas et al., 2007). One particular study, regarding daily box office performance for recently released movies, concluded that online review ratings had no influence in future daily revenues, suggesting that the evaluations of other users had no persuasive effect over consumers’ picks of movie(s) to watch. However, the same study confirmed that the increased volume of online posting alone had a positive impact on revenues due to the awareness effect, as online reviews are an indicator of intensity of word of mouth, which is a significant driver for movie sales (Duan et al., 2008). Chevalier and Mayzlin (2006) deserve a special mention for their pioneer analysis of user reviewing behaviour, through the observation of aggregate review scores and sales figures across 2 different online book stores (Amazon.com and Barnes and Noble’s website bn.com) over the course of a year. They concluded that changes in user reviews had a significant effect on sales, not only if you consider fluctuations in average review scores, but also based on the percentage of negative opinions. The study went as far as analysing the length of commentaries in user reviews to support the idea that consumers read and respond to user generated content and take it into account in the decision-making process. Since that pioneer work, several studies have applied the same principles to different markets, the main consensus being that online shoppers adopt online reviews as reliable information sources in their decision making – see Filieri & McLeay (2014) for an application to travel and accommodation markets, and Cui et al., (2012) for research on consumer electronics and videogames. 13 vii. Customer satisfaction theories Yi's (1990) review on the constructs behind customer satisfaction theory is commonly referenced for presenting multiple propositions for the measurement and evaluation of consumer (dis)satisfaction. Most of those studies revolve around the disconfirmation theory, according to which consumers evaluate their products by comparing their pre-purchase expectations with the actual performance perceived during the consumption experience (Oliver, 1980). When that experience is as pleasurable as, or more pleasurable than, their expectations, consumers are left with a sense of satisfaction; whereas if the emotional outcome of consumption is subpar to expectations, then consumers will be dissatisfied. Two of the most commonly accepted theories differ only on the basis by which those standards or expectations are created:  the Comparison level theory is built on the idea that expectations are influenced by three basic sources: the consumer’s past experiences with similar products; the situational context in which a product is offered, be it advertising or promotional efforts by manufacturers; and their knowledge of other consumers’ experiences with the product under evaluation. Each instance of consumption will “force” the consumers to adapt their expectations by altering their comparison level (LaTour & Peat, 1979)  the Value perception disparity theory implies that pre-formed expectations for products are an insufficient referential for customer satisfaction measurement, since expectations may not accurately represent an individual’s needs, wants and desires. For example, individuals may create expectations for specific functions/features of a product but not others, while in practice those “extra” functions add to the consumer’s perceived value of the product and therefore can increase purchase satisfaction. Therefore, the goodness of fit between the consumer’s values and the objective performance of the product should instead be used to evaluate CS (Westbrook & Reilly, 1983). Despite there being studies that support the validity of both hypotheses, neither comparison levels nor value perception disparities are sufficient to fully explain the phenomenon of consumer satisfaction by themselves (Yi, 1991). Instead, those and other 14 constructs, such as transactional equity (Fisk & Young, 1985) and normative expectations (Woodruff et al., 1983), should be used complementarily. Whereas satisfaction is thought of as an affective outcome of consumption, and as such being in essence a post-purchase construct, more recent studies focus on the cognitive process by which consumers assess the utility, or value, of a product, based on the sacrifices and rewards associated with its purchase and/or consumption (Zeithaml, 1988). The customer perceived value theory has at its core the idea of a trade-off between quality and price, and proposes that consumers make their purchasing decisions based on the goal of maximizing the value-for-money ratio (Cravens et al., 1988). To evaluate the factors that contribute to the perception of value by customers, Sweeney & Soutar (2001) developed PERVAL, a four-dimensional scale that accounts for the effects of:  Emotional value – feelings and affectional states derived from consumption;  Social value – the product’s potential to enhance the individual’s notion of its own social standing;  Price – perception of sacrifices incurred on the acquisition of products, either through objective costs or costs of opportunity;  Quality – functional value as measured by the product’s performance. The weight of each of these dimensions is implied to vary from consumer to consumer (Zeithaml, 1988), product to product (Sheth, 1991) and even at different decision levels – buy/not buy, choice of brand, choice of product. Therefore, multidimensional scales of product performance, quality and satisfaction are common, both for retail products and for services (SERVQUAL – Parasuraman et al., 1988). These theoretical foundations still guide current literature and hold true for recent empirical applications of consumer satisfaction constructs (Flint et al., 2011; Malik 2012). 15 viii. Systems for Online User Reviews Online product reviews happen when consumers articulate their thoughts on product quality and post-purchase satisfaction through the Internet. To accurately portray the results of online WoM, several different types of reviewing systems can be used. Amazon.com’s model, exemplified in Figure 2, allows users to rate the products they purchased on a 5-point scale, and results are most readily presented as the average score, as well as the absolute frequency of each discrete evaluation. Figure 2 - Example of Amazon product review summary Booking.com, a prominent accommodation booking website, encourages its customers to evaluate hotels based on several dimensions (location, comfort, price-quality relation, staff, etc.) and agglomerates those results into a simple average score as seen in Figure 3. Figure 3 - Example of booking.com review system Buyers and sellers on ebay.com evaluate each other by the quality of their interaction during a transaction, on a 3-option scale: the experience was either negative, neutral or positive. The seller’s reputation is based on its past buyers’ feedback and is 16 presented in the product page as an aggregate feedback score, as well as the percentage of positive feedback, in an effort to inspire trust in future buyers – see Figures 4 and 5. Figure 4 - ebay.com seller panel on a product page (example) Figure 5 - ebay.com page for the same seller as in Figure 4 In the first two cases, as well as the seller page for ebay.com, customers are asked to evaluate their perception of quality in the products purchased or the seller trustworthiness, not unlike the traditional method of measuring satisfaction through Likert scales. These reviewing systems allow future buyers to select the products or sellers that best fit their own values, or in other words, the features they value the most. In other cases, such as the Steam platform, consumer reviews are made on a twooption scale: users either recommend a product they purchase, or recommend against it. The outcome is then aggregated and translated into a 9-point scale, ranging from 17 “Overwhelmingly Negative” to “Overwhelmingly Positive” based on the total number of reviews and the percentage of reviews that are positive. This percentage is also shown on the platform’s product search engine and can be used as a sorting criteria to order products from the most recommended to least recommended. The absolute number of positive and negative reviews can also be seen on the product page – see Figure 6. Figure 6 - Example of Steam Store Product Page, with the Product Reviews information highlighted Figure 7 - Example of Customer Reviews section in a Steam product page 18 Users can also rate customer reviews on their helpfulness and/or humorous nature, which allows the platform to highlight the posts with the best feedback on each of those criteria, as well as order them by publication date – Figure 7 shows an example of a Product review where these attributes can be seen. The Steam reviewing system invokes the notion of a certain standard of quality that is pre-formed by consumers based on price and quality perceptions and that generate feelings of satisfaction or discomfort when compared to the actual perceived quality of products post-consumption. To understand this mechanism, it is particularly useful to analyse the thesis by Voss et al. (1998) which, although primarily focused on service exchanges, is among the first articles to emphasize the role of pricing in predicting customer satisfaction. 25 iv. Main variables for research To test the hypotheses presented at the start of this chapter, we will be analysing the evolution of two main variables over a period of 29 daily observations (14 days before to 14 days after the first occurrence of a price discount):  New reviews published daily. Considering that we are able to extract the total number of reviews existing at the time of extraction: RT: Total number of user reviews existing on day T rT: Number of new reviews published on day T Then: (1) RT = RT−1 + rT  Average product review scores: is defined by the percentage of all reviews posted up until the time of extraction that are positive: ST: Average review score on day T, i.e., percentage of all reviews on day T that are positive PT: Total number of positive reviews existing on day T pT: New positive reviews published on day T (2) ST = PT RT = PT−1+ pT RT−1+ rT Given the simplicity in the nature of these variables, Expression 2 means that, mathematically, in order for there to be an increase in the review scores on day T, i.e., ST > ST−1 it must be true that: pT rT> PT−1 RT−1 26 Hence, the data collected should be sufficient to detect periods in which the average score of reviews posted was higher (or lower) than the sum of all reviews posted up until that point in time. The serially dependent nature of these variables defies the assumption of i.i.d. observations for the Change Point Analysis method that will be used to detect changes in means (tests for autocorrelation can be found in Appendix I). This will lead to an increased vulnerability to type I errors because of the possibility of overestimation of residuals (Lund et al., 2007). However, this assumption is one that is frequently overlooked by literature in the analysis of empirical data, since real time series are rarely stationary and truly stochastic processes. 27 v. Change Point Analysis and CUSUM charts Change point analysis is a statistical method commonly used in several fields, from Biology to Finance, which tests for the occurrence of changes in the parameters of a distribution in a series of time-ordered observations (Matteson et al., 2012). This method has its bases on Page's (1955) proposition of a new method for detecting changes in the mean of an observation through Cumulative Sum Control Charts. By comparing each T observation with the average value of the series, Page proposed a function: CT: Cumulative sum of the differences to the series’ average (3) CT={0, T=0 CT−1 +(XT− X ), T>0 that can be represented by a CUSUM chart. In segments of the time series where the values of the observations are above the overall series average, the chart will have a positive slope. If a change point occurs, and the parameter of the distribution that is being tested for suddenly changes, then the slope of the function will change as well (Taylor, 2000). In the occurrence of a change point, it is to be expected that the time series is arranged in such a way that the CUSUM chart is similar to a U or V-shape, or its inverse. The method of change point analysis takes this concept one step further. Firstly, we calculate Cdiff = Cmax − Cmin , which gives us the difference between the highest and lowest values of the CUSUM function. Then, a set of bootstrap samples are generated by randomly rearranging the time series observations, in order to simulate what the CUSUM function should look like if no change point is present in the series. The hypothesis of the occurrence of a change point is tested by comparing the original series with the bootstraps generated, with the confidence level of the test being equivalent to the percentage of bootstraps with Cdiff higher than the original series’Cdiff. The same method can be used to test for changes in variation. This test is also robust in the presence of outliers, which is particularly useful given the existence of missing data entries in our sample dataset. 28 Because of the exploratory nature of this study and the lack of sufficiently proved knowledge on how our variables behave in nature, we opted for a nonparametric method of change point analysis for our tests, making as little assumptions about the distribution as possible. Additionally, because we’re testing for multiple time series (specifically, one for each of the products of our sample), we needed a test capable of multivariate time series analysis. The statistical package that presented the best fit for our needs was the ecp R package – see James & Matteson (2014) for an explanation of the computational process for the hierarchical divisive estimation tests. 29 IV. Results and Findings i. New reviews published daily by product Graphically representing and analysing a data set of 658 x 29 entries is not an easy task without the recourse to statistical software packages such as R. The plot for reviews published daily, however, is a simple one to reproduce and, to an extent, interpret. Figure 9 depicts the evolution of this variable, where it is easy to detect that an outstanding number of reviews are published in the few days after the start of a price discount (t=15), when compared with the relatively stable process of generation of user recommendations. Figure 9 - Reviews Published Daily by Product Testing for a change point (see Appendix II for the computer-assisted statistical test) gives a positive result for a change in means in t=16 with at least 99% confidence. Thus, we confirm Hypothesis 1, and conclude, about the behaviour of consumers in the Steam platform, that a price reduction has a significant impact on the volume of product reviews published in the days following the pricing initiative. 30 ii. Average review scores by product For review scores, the visualization of data was a more complex challenge. In our first attempt to explore the data in our sample, we built the time series chart based on the average review scores for those products. The result, shown in Figure 10, provided some information about the behaviour of our variables. Figure 10 – Time Series plot for Review Scores One of the most evident interpretations of this plot is that average review scores are relatively stable for the period of analysis, which is coherent with the fact that we’re analysing cumulative, serially dependent scores. Out data is also presented in a percentpoint scale, which explains the predominantly parallel lines. This chart was certainly not sufficient to answer our research questions, although it was immediately apparent that review scores are subject to more frequent variations after t=15 (incidentally, the first day of the price discount), as we can observe from the higher density of segments with non-null slope in the second half of the time series plot. Although review scores range from 12% to 100% for the period observed, they are highly concentrated around the 85% mark and the distribution is relatively stable, as we can observe through the time series boxplot in Figure 11 and the statistical data in Table 2. 31 *t=1 affected by missing values Figure 11 - Time Series Boxplot for Product Review Scores DISTRIBUTION OF REVIEW SCORES Stats t=1 t=15 t=29 10th Percentile 59% 58% 59% 25th Percentile 73% 73% 73% 50th Percentile 85% 85% 85% 75th Percentile 92% 92% 92% 90th Percentile 95% 95% 95% 𝑿  80,493% 80,234% 80,365% 𝝈𝟐 0,0239 0,0239 0,0234 𝝈 0,1546 0,1546 0,1531 *revised to account for missing values Table 2 - Distributional statistics for Product Review Scores 32 To visually interpret our data, we found it useful to compare the review score of each product at a certain moment in time, St, to the review score it had on the moment of occurrence of the discount (S15, which is its score for our standard date t=15; see the definition of this concept in Section III.iii – Sample Selection and Data Treatment). For that matter, we create yet another construct – the Standard Score ( S′t ): (4) S′t = St S15 The graphical representation for this modified variable, results in time series chart in Figure 12: Figure 12 - Standardized Review Scores plot This new plot seems to confirm that after the occurrence of a discount (t=15) there is a higher volatility in customer reviews scores than before, and that in that case the signal of variation can be either positive or negative. The first step towards applying our change point analysis methodology was to build the CUSUM chart for the evolution of review scores, reproduced in Figure 13. 33 Figure 13 - CUSUM Chart for Review Scores (Means) In section III.v we described what a CUSUM control chart would look like for a time series where a change point is present – a U or V shape (or its inverse) with the maximum absolute value located in or around the change point. Looking at our data it is easy to recognize a similar pattern for a considerable number of time series (products). The final step is to run the test for a change point of the distributional means with the ecp package – see Appendix III for the programmatic testing process – which returns positive results for a first-level change point at t=17 with at least 99% confidence, equivalent to dividing the time series into two clusters, t=[1, 17[ and t=[17, 29] with statistically significant differences in means. This result is coherent with the previous analysis of the CUSUM control chart, which is noticeably skewed towards the end of the time series (and to the right of t=15, the first observation with presence of a price discount). This can easily be justified because of: a) the delay between the moment a discount is activated and the moment a consumer finally publishes his review, after effectively acquiring and having 34 a first experience with the product, which we detected in the section regarding the analysis of new reviews to be at least 1 day; b) The serial dependence of the variable in cause, paired with some rigidity in the scale (product ratings are presented in percentages with no decimal places) which provides resistance and causes a delay in variations of average review scores. The confirmation of a statistically significant change point in the time series around the occurrence of a price discount is deemed sufficient to accept Hypothesis 2. Thus we conclude that, regarding review scores for products in the Steam Storefront, the occurrence of a price discount causes a shift in the means of the distribution. Regarding the signal of that shift, analysing a sub-sample of products that have review information available for both t=14 (last observation before a discount) and t=29:  103 products for which S14 > S29  325 products for which S14 = S29  83 products for which S14 < S29 41  The possibility of a correlation between discount rates and the impact of the discount in product reviews could allow for more precise pricing actions, making it easier to forecast increases in sales in function of the percentage of price reduction.  Understanding if different attributes of products can potentially be correlated with review scores (and the variations in review scores caused by pricing changes) could potentially identify clusters of products that are more or less susceptible to customer backlash. Understanding, for example, if a certain genre of video game has higher price-review elasticity would allow for the design of specific pricing strategies for those products.  The timing of occurrence of a discount could also influence its effectiveness. As we’ve seen in this study, there is a higher volume of product reviews published on weekend than weekdays. Understanding that dynamic could lead to more cost-effective pricing actions. Furthermore, it could prove useful to use discourse analysis methods to observe changes in the emotional and affectional cues in textual product reviews. The interpretation of online shoppers’ verbalizations of product quality and satisfaction could allow for a deeper understanding of the consumer’s behavioural processes, revealing factors and variables that can’t be observed exclusively through review scores. In sum, we believe that our research was a comprehensive first step for setting the foundations of an innovative and exciting approach of e-commerce strategies and electronic word of mouth marketing. In the form of an extensive literature review on the constructs behind social commerce and online recommendation systems, and an empirical examination of online marketplace behaviour, our main contribution was to offer a basis and directions for future research on a topic that is increasingly relevant and impactful. 42 VII. References Anderson, E. W. (1998), "Customer Satisfaction and Word of Mouth", Journal of Service Research, Vol. 1, Nr. 1, pp. 5–17. Arndt, J. (1967), "Role of Product-Related Conversations in the Diffusion of a New Product", Journal of Marketing Research, Vol. 4, Nr. 3, pp. 291. Balasubramanian, S. and V. Mahajan (2001), "The Economic Leverage of the Virtual Community", International Journal of electronic Commerce, Vol. 5, Nr. 3, pp. 103–138. Bambauer-Sachse, S. and S. Mangold (2011), "Brand equity dilution through negative online word-of-mouth communication", Journal of Retailing and Consumer Services, Vol. 18, Nr. 1, pp. 38–45. Barlow, J. P. (1995), "Is There a There in Cyberspace?", Utne Reader, pp. 53–56. Blodgett, J. G. D. H. Granbois and R. G. Walters (1993), "The effects of perceived justice on complainants’ negative word-of-mouth behavior and repatronage intentions", Journal of Retailing, Vol. 69, Nr. 4, pp. 399–428. Burnett, G. (2000), "Information exchange in virtual communities: A typology", Information Research, Vol. 5, Nr. 4. Chen, Y. and J. Xie (2008), "Online Consumer Review: Word-of-Mouth as a New Element of Marketing Communication Mix", Management Science, Vol. 54, Nr. 3, pp. 477–491. Cheung, C. M. K. and M. K. O. Lee (2012), "What drives consumers to spread electronic word of mouth in online consumer-opinion platforms", Decision Support Systems, Vol. 53, Nr. 1, pp. 218–225. Cheung, R. (2014), "The Influence of Electronic Word-of-Mouth on Information Adoption in Online Customer Communities.", Global Economic Review, Vol. 43, Nr. 1, pp. 42–57. Chevalier, J. and D. Mayzlin (2006), "The Effect of Word of Mouth on Sales: Online Book Reviews", Journal of Marketing Research, Vol. 43, Nr. 3, pp. 345–354. The Nielsen Company (2013), Global Trust in Advertising and Brand Messages Is Key in Advertising, September 2013. Cravens, D. W., C. W. Holland, C. W. Lamb and W. C. Moncrief (1988), "Marketing’s role in product and service quality", Industrial Marketing Management, Vol. 17, Nr. 4, pp. 285–304. 43 Cui, G., H. Lui and X. Guo (2012), "The Effect of Online Consumer Reviews on New Product Sales", International Journal of Electronic Commerce, Vol. 17, Nr. 1, pp. 39–58. Darke, P. R. and C. M. Chung (2005), "Effects of pricing and promotion on consumer perceptions: It depends on how you frame it", Journal of Retailing, Vol. 81, Nr. 1, pp. 35–47. Dellarocas, C., X. Zhang and N. F. Awad (2007), "Exploring the value of online product reviews in forecasting sales: The case of motion pictures", Journal of Interactive Marketing, Vol. 21, Nr. 4, pp. 23–45. Dichter, E. (1966), "How Word of Mouth Advertising Works", Harvard Business Review, pp. 147–166. Duan, W., B. Gu and A. B. Whinston (2008), "Do online reviews matter? - An empirical investigation of panel data", Decision Support Systems, Vol. 45, Nr. 4, pp. 1007– 1016. Engel, J. F., R. J. Kegerreis and R. D. Blackwell (1969), "Word-of-mouth Communication by the Innovator.", Journal of Marketing, Vol. 33, Nr. 3, pp. 15– 19. Eryarsoy, E. and S. Piramuthu (2014), "Experimental evaluation of sequential bias in online customer reviews", Information & Management, Vol. 51, Nr. 8, pp. 964– 971. Falk Moore, S. and B. Myerhoff (1975), "Prologue", in Symbol and Politics in Communal Ideology, pp. 13–23, Ithaca, Cornell University Press. Festinger, L. (1957), "A theory of cognitive dissonance", Scientific American, Vol. 207. Filieri, R. and F. McLeay (2014), "E-WOM and Accommodation: An Analysis of the Factors That Influence Travelers’ Adoption of Information from Online Reviews", Journal of Travel Research, Vol. 53, pp. 44–57. Fisk, R. and C. Young (1985), "Disconfirmation of Equity Expectations: Effects on Consumer Satisfaction With Services", Advances in Consumer Research, Vol. 12, pp. 340–345. Flint, D. J., C. P. Blocker and P. J. Boutin (2011), "Customer value anticipation, customer satisfaction and loyalty: An empirical examination", Industrial Marketing Management, Vol. 40, Nr. 2, pp. 219–230. Frenzen, J. and K. Nakamoto (1993), "Structure, Cooperation, and the Flow of Market Information", Journal of Consumer Research, Vol. 20, Nr. 3, pp. 360. 44 Hajli, N. (2015), "Social commerce constructs and consumer’s intention to buy", International Journal of Information Management, Vol. 35, Nr. 2, pp. 183–191. Harrison, S. and P. Dourish (1996), "Re-Place-ing Space : The Roles of Place and Space in Collaborative Systems", Proceedings of the 1996 ACM conference on Computer supported cooperative work, pp. 67–76. Hennig-Thurau, T., K. P. Gwinner, G. Walsh and D. D. Gremler (2004), "Electronic word-of-mouth via consumer-opinion platforms: What motivates consumers to articulate themselves on the Internet?", Journal of Interactive Marketing, Vol. 18, Nr. 1, pp. 38–52. Herring, S. C. (2001), "Computer-Mediated Discourse", Discourse, January issue, pp. 1– 24. Hu, N., L. Liu and J. J. Zhang (2008), "Do online reviews affect product sales? The role of reviewer characteristics and temporal effects", Information Technology and Management, Vol. 9, Nr. 3, pp. 201–214. Jacoby, J. and D. B. Kyner (1973), "Brand Loyalty Vs. Repeat Purchasing Behavior", Journal of Marketing Research, Vol. 10, Nr. 1, pp. 1–9. James, N. A. and D. S. Matteson (2014), "ecp: An R Package for Nonparametric Multiple Change Point Analysis of Multivariate Data", Journal of Statistical Software, Vol. 62, Nr. 7. Jöckel, S., A. Will and F. Schwarzer (2008), "Participatory Media Culture and Digital Online Distribution—Reconfiguring the Value Chain in the Computer Game Industry", International Journal on Media Management, Vol. 10, Nr. 3, pp. 102– 111. Jones, Q. and S. Rafaeli (2000), "Time to Split , Virtually: “Discourse Architecture” and “Community Building” Create Vibrant Virtual Publics", Electronic Markets, Vol. 10, Nr. 4, pp. 214–223. Ke-Wei, H. and A. Sundararajan (2011), "Pricing Digital Goods: Discontinuous Costs and Shared Infrastructure", Information Systems Research, Vol. 22, Nr. 4, pp. 721–738. Kiesler, S., J. Siegel and T. W. McGuire (1984), "Social psychological aspects of computer-mediated communication.", American Psychologist, Vol. 39, Nr. 10, pp. 1123–1134. Kleinberg, R. and T. Leighton (2003), "The value of knowing a demand curve: bounds on regret for online posted-price auctions", Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. 45 Komito, L. (1998), "The Net as a Foraging Society: Flexible Communities", The Information Society, Vol. 14, Nr. 2, pp. 97–106. LaTour, S. A. and N. C. Peat (1979), "Conceptual and Methodological Issues in Consumer Satisfaction Research", Advances in Consumer Research, Vol. 6, Nr. 1, pp. 431–437. Lee, M. and S. Youn (2009), "Electronic word of mouth (eWOM): How eWOM platforms influence consumer product judgement", International Journal of Advertising, Vol. 28, Nr. 3, pp. 473–499. Li, X. and L. M. Hitt (2010), "Price Effects in Online Product Reviews: An Analytical Model and Empirical Analysis", MIS Quarterly, Vol. 34, Nr. 4, pp. 809. Lund, R., X. L. Wang, Q. Q. Lu, J. Reeves, C. Gallagher and Y. Feng (2007), "Changepoint detection in periodic and autocorrelated time series", Journal of Climate, Vol. 20, Nr. 20, pp. 5178–5190. Malik, S. U. (2012), "Customer Satisfaction, Perceived Service Quality and Mediating Role of Perceived Value", International Journal of Marketing Studies, Vol. 4, Nr. 1, pp. pp. 68–76. Matteson, D. S. and N. A. James (2012), "A nonparametric approach for multiple change point analysis of multivariate data", Journal of the American Statistical Association, Vol. 109, Nr. 505, pp. 334-345. Nambisan, P. and J. H. Watt (2011), "Managing customer experiences in online product communities", Journal of Business Research, Vol. 64, Nr. 8, pp. 889–895. Page, E. S. (1955), "A Test for a Change in a Parameter Occurring at an Unknown Point", Biometrika, Vol. 42, Nr. 3, pp. 523–527. Parasuraman, A., V. A. Zeithaml and L. L. Berry (1988), "SERVQUAL : A MultipleItem Scale for Measuring Consumer Perceptions of Service Quality", Journal of Retailing, Vol. 64, Nr. 1, pp. 12–40. Pavlou, P. A. and D. Gefen (2004), "Building effective online marketplaces with institution-based trust", Information Systems Research, Vol. 15, Nr. 1. Peddibhotla, N. B. and M. R. Subramani (2007), "Contributing to Public Document Repositories: A Critical Mass Theory Perspective", Organization Studies, Vol. 28, Nr. 3, pp. 327–346. Porter, C. E. (2004), "A Typology of Virtual Communities: A Multi-Disciplinary Foundation for Future Research", Journal of Computer-Mediated Communication, Vol. 10. 46 Price, L. L., L. F. Feick and A. Guskey (1995), "Everyday Market Helping Behavior", Journal of Public Policy & Marketing, Vol. 14, Nr. 2, pp. 255–266. PwC Global (2015), "Total Retail 2015:", February 2015. Rafaeli, S. and F. Sudweeks (1997), "Networked Interactivity", Journal of ComputerMediated Communication, Vol. 2, Nr. 4. Richins, M. L. (1984), "Word of Mouth Communication as Negative Information", Advances in Consumer Research, pp. 697–702. Ridings, C. M. and D. Gefen (2004), "Virtual Community Attraction: Why People Hang Out Online", Journal of Computer-Mediated Communication, Vol. 10. Schafer, J. B., J. Konstan and J. Riedi (1999), "Recommender systems in e-commerce", Proceedings of the 1st ACM conference on Electronic commerce EC 99, 2001, pp. 158–166. Schlossberg, H. (1991), "Customer Satisfaction: Not a fad, but a way of life", Marketing News, Vol. 25, Nr. 20, pp. 18. Schouten, J. W. and J. H. McAlexander (1995), "Subcultures of Consumption: An Ethnography of the New Bikers", Journal of Consumer Research, Vol. 22, Nr. 1, pp. 43. See-To, E. and K. Ho (2014), "Value co-creation and purchase intention in social network sites: The role of electronic Word-of-Mouth and trust – A theoretical analysis", Computers in Human Behavior, Vol. 31, pp. 182–189. Senecal, S. and J. Nantel (2004), "The influence of online product recommendations on consumers’ online choices", Journal of Retailing, Vol. 80, Nr. 2, pp. 159–169. Shen, W., Y. J. Hu and J. Rees (2015), "Competing for Attention : An Empirical Study of Online Reviewers ’ Strategic Behaviors", MIS Quarterly, Vol. 39, Nr. 3, pp. 683-696. Shen, X. L., K. Zhang and S. J. Zhao (2014), "Understanding information adoption in online review communities: The role of herd factors", Proceedings of the Annual Hawaii International Conference on System Sciences, 2014, pp. 604–613. Sheth, J. N., B. I. Newman and B. L. Gross (1991), "Why we buy what we buy: A theory of consumption values", Journal of Business Research, Vol. 22, Nr. 2, pp. 159– 170. Solomon, G. and L. Schrum (2007), Web 2.0: New Tools, New Schools, ISTE, pp. 7–24. Sundaram, D. S. and B. Hills (1998), "Word-of-Mouth Communications : A Motivational Analysis", Advances in Consumer Research, Vol. 25, pp. 527–531. 47 Swan, J. E. and R. L. Oliver (1989), "Postpurchase Communications by Consumers", Journal of Retailing, Vol. 65, Nr. 4, pp. 516–533. Sweeney, J. C. and G. N. Soutar (2001), "Consumer perceived value: The development of a multiple item scale", Journal of Retailing, Vol. 77, Nr. 2, pp. 203–220. Tayebi, A. (2013), "“Communihood:” A Less Formal or More Local Form of Community in the Age of the Internet", Journal of Urban Technology, Vol. 20, Nr. 2, pp. 77– 91. Taylor, W. A. (2000), "Change-point analysis: a powerful new tool for detecting changes", available at http://www. variation. com/cpa/, accessed 15 September 2015. Tonnies, F. (1957), Community and Association, London, Routledge and Kegan Paul. Tybout, A. M., B. J. Calder and B. Sternthal (1981), "Using Information Processing Theory to Design Marketing Strategies", Journal of Marketing Research, Vol. 18, Nr. 1, pp. 73–79. Voss, G. B., A. Parasuraman and D. Grewal (1998), "The Roles of Price, Performance, and Expectations in Determining Satisfaction in Service Exchanges", Journal of Marketing, Vol. 62, Nr. 4, pp. 46–61. Walther, J. B. (1996), "Computer-Mediated Communication: Impersonal, Interpersonal, and Hyperpersonal Interaction", Communication Research, Vol. 23, Nr. 1, pp. 3– 43. Weinberg, B. D., K. Ruyter, C. Dellarocas, M. Buck and D. I. Keeling (2013), "Destination social business: exploring an organization’s journey with social media, collaborative community and expressive individuality", Journal of Interactive Marketing, Vol. 27, Nr. 4, pp. 299–310. Wellman, B. (1997), "An Electronic Group is Virtually a Social Network", in Culture of the Internet, Kiesler, S., New Jersey: Lawrence Erlbaum, pp. 179–205. Westbrook, R. and M. Reilly (1983), "Value-percept Disparity: an Alternative to the Disconfirmation of Expectations Theory of Consumer Satisfaction", Advances in Consumer Research, Vol. 10, pp. 256–261. Wild, R. (1981), Australian Community Studies and Beyond, Sydney: Allen & Unwin. Woodruff, R. B., E. R. Cadotte and R. L. Jenkins (1983), "Modeling consumer satisfaction processes using experience-based norms", Journal of Marketing Research, Vol. 20, pp. 296–304. Yi, Y. (1990), "A critical review of consumer satisfaction", Review of Marketing, Vol. 4, Nr. 1, pp. 68–123. 48 Zeithaml, V. A. (1988), "Consumer Perceptions of Price, Quality, and Value: A MeansEnd Model and Synthesis of Evidence", Journal of Marketing, Vol. 52, July issue, pp. 2–22. 49 VIII. Appendices Appendix I – Autocorrelation Factors for Number of Reviews R script # Loading Time Series TotRevByDate = read.xls("TotRevByDate.xlsx", header=F) NewRevByDate = read.xls("NewRevByDate.xlsx", header=F) ## Convert Data into Time Series ts(TotRevByDate) ts(NewRevByDate) ### Test for Autocorrelation on Total Reviews par(mfrow=c(1,2)) acf(TotRevByDate) pacf(TotRevByDate) ### Test for Autocorrelation on New Reviews par(mfrow=c(1,2)) acf(NewRevByDate) pacf(NewRevByDate) Autocorrelation factors – ACF/PACF plots 1 – ACF and PACF tests for Total Reviews by Date, showing a significant first degree lag in the observations, indicating high levels of autocorrelation. 2 – ACF and PACF tests for New Reviews by Date, showing significant first degree lag in the observations, indicating high levels of autocorrelation, as well as a significant periodicity, likely referring to the higher number of reviews posted on weekends in comparison to weekdays (6 days partial lag). 50 Time series – Total Reviews By Date (“TotRevByDate.xlsx”) > ts(TotRevByDate) Time Series: Start = 1 End = 71 Frequency = 1 V1 [1,] 2854016 [2,] 2864083 [3,] 2866254 [4,] 2872106 [5,] 2873828 [6,] 2884733 [7,] 2891389 [8,] 2899837 [9,] 2906390 [10,] 2914368 [11,] 2919066 [12,] 2926101 [13,] 2934424 [14,] 2934623 [15,] 2946753 [16,] 2950592 [17,] 2959433 [18,] 2967977 [19,] 133620 [20,] 2965740 [21,] 2988006 [22,] 2998956 [23,] 3005991 [24,] 3012173 [25,] 3015864 [26,] 3014618 [27,] 3023517 [28,] 3019920 [29,] 3040268 [30,] 3049826 [31,] 3055296 [32,] 3069293 [33,] 3069293 [34,] 3076561 [35,] 3086038 [36,] 3091155 [37,] 3097408 [38,] 3112225 [39,] 3112225 [40,] 3126267 [41,] 3132158 [42,] 3146781 [43,] 3148806 [44,] 3137864 [45,] 3152318 [46,] 3162114 [47,] 3170345 [48,] 3187136 [49,] 3187136 [50,] 3193085 [51,] 3198398 [52,] 3201511 [53,] 3198829 [54,] 3210809 [55,] 3226889 [56,] 3208214 [57,] 3240210 [58,] 3225212 [59,] 3264419 [60,] 3271357 [61,] 3278192 [62,] 3288676 [63,] 3252233 [64,] 3298631 [65,] 3304130 [66,] 3295189 [67,] 3319577 [68,] 3328329 [69,] 3332906 [70,] 3333364 [71,] 3339912