Optimal Aggregation of Consumer Ratings

OPTIMAL AGGREGATION OF CONSUMER RATINGS ABSTRACT Consumer review websites such as Yelp.com leverage the wisdom of the crowd, with each product being reviewed many times (some with more than 1000 reviews). Because of this, the way in which information is aggregated is a central decision faced by consumer review websites. Given a set of reviews, what is the optimal way to construct an average rating? We offer a structural approach to answering this question, allowing for (1) reviewers to vary in stringency (some reviewers tend to leave worse reviews on average) and accuracy (some reviewers are more erratic than others), (2) reviewers to be influenced by existing reviews, and (3) product quality to change over time. We apply this approach to reviews from Yelp.com to derive optimal ratings for each restaurant (in contrast with the arithmetic average displayed by Yelp). Because we have the history of reviews for each restaurant and many reviews left by each reviewer, we are able to identify these factors using variation in ratings within and across reviewers and restaurants. Using our estimated parameters, we construct optimal ratings for all restaurants on Yelp, and compare them to the arithmetic averages displayed by Yelp. As of the end of our sample, a conservative finding is that roughly 25-27% of restaurants are more than 0.15 stars away from the optimal rating, and 8-10% of restaurants are more than 0.25 stars away from the optimal rating. This suggests that large gains could be made by implementing optimal ratings. Much of the gains come from our method responding more quickly to changes in a restaurant’s quality. Our algorithm can be flexibly applied to many different review settings. 1 Introduction The digital age has transformed the way that consumers learn about product quality. Websites ranging from Yelp and TripAdvisor to eBay and Amazon use crowdsourcing to generate product ratings and reviews. This has dramatically increased the amount of information consumers have when making a decision. The value of this information increases in the number of reviews being left. However, the more reviews that are left, the more time-consuming and di⇥cult it becomes for a consumer to process the underlying information. This calls for the platform to generate an easy-to-understand metric that summarizes existing reviews on a specific subject. In this paper, we develop a method to analyze and aggregate reviews, and apply this method to restaurant reviews from Yelp.com. How should consumer review websites present information to readers? In principle, one could simply present all of the underlying reviews and allow consumers to decide for themselves how to aggregate information. In fact, there are some websites that do this, and hence avoid the need to aggregate information. Yet a growing literature has demonstrated that the impact of information depends not only on the informational content but also on the salience and simplicity of the information (Brown et al 2010, Luca and Smith forthcoming, Pope 2009). In the case of Yelp, for instance, consumers respond directly to the average rating even though this is coarser than the underlying information (Luca 2011). Because of consumer inattention, the method chosen to aggregate information is of first-order importance. Currently, many review websites (including Yelp) present an arithmetic mean of all reviews written for a given product. Implicitly, this method of aggregation treats each review as an equally informative noisy signal of quality. In other words, arithmetic average is only optimal under very restrictive conditions - such as when reviews are unbiased, independent, and identically distributed signals of true quality. The goal of this paper is to move toward optimal aggregation of consumer reviews. We consider an aggregate rating to be optimal if two conditions are met. First, observable preferences and biases of different types of consumers must be separated from a consumer’s vertical signal of quality. Second, reviews must be weighted to account for informational content, with more weight endogenously assigned to reviews containing more information. This includes both the fact that some reviewers may be more accurate than others and the fact that product quality may change over time. The product of this paper is a single aggregated measure of vertical quality. To derive an optimal aggregation algorithm, we develop a structural framework that allows reviewers to vary in accuracy (some reviewers are more erratic than others), stringency (some reviewers leave systematically lower ratings), and reputational concerns (some reviewers care more about reputation on Yelp than others, and hence prefer not to deviate from prior reviews). Our framework also accounts for the fact that a restaurant’s quality can change over time, which implies that concurrent restaurant quality is better reflected in recent reviews than in early reviews. 1 Because we have the entire history of reviews for each restaurant and many reviews left by each reviewer, we are able to identify these factors using variation in ratings within and across reviewers and restaurants. For example, stringency of a reviewer can be identified using restaurant and reviewer-type fixed effects. Similarly, accuracy of a reviewer can be identified by variation in how far different reviewers are (in expectation) from the long-run average rating of the restaurants they review. To identify changes in restaurant quality, we impose the assumption that the evolution of restaurant quality follows a martingale process by calendar time, and estimate the underlying parameters. Our model also allows restaurant ratings to follow a common time trend since the first Yelp review of a restaurant, which could capture a linear trend of reviewer stringency relative to the first review of the same restaurant, or a linear trend of true quality in addition to the martingale evolution of quality. Using our estimated parameters, we then construct optimal average ratings for each restaurant on Yelp, and compare them to the simple arithmetic mean by Yelp. The results depend on how we interpret a significant downward trend of ratings within a restaurant. If this “chilling” effect is interpreted as reviewer bias only (relative to the first review), we find that, by the end of the sample, more than half of restaurants have their Yelp-style simple average ratings differ from the optimal by more than 0.15 stars, and more than one-quarter of restaurants have Yelp-style average ratings differ from the optimal by more than 0.25 stars. If the above chilling effect is interpreted as changes in true quality, the absolute difference between simple and optimal average ratings is still more than 0.15 stars for 25-27% of restaurants, and more than 0.25 stars for 8-10% of restaurants by the end of the data sample. Most of the optimal-vs-simple-average difference is driven by evolution of restaurant quality. This is because the simple average weights a restaurant’s first review the same as it weights the thousandth review. In contrast, our algorithm reduces the weight assigned to early reviews and hence more quickly adapts to changes in quality. Reviewer reputation, on the other hand, has little impact on the optimal average in the Yelp setting, even though reputational concerns may be an important part of the decision to become a Yelp reviewer to begin with. For example, "elite" status is a designation given by Yelp to prolific reviewers, who leave what Yelp deems to be higher quality reviews.1 Our model shows that elite and non-elite reviewers have different signal precision and reputational concerns. Elite reviewers provide ratings with higher precision; these ratings are also closer to a restaurant’s long-run average rating. Moreover, estimates suggest that elite reviewers are more likely to incorporate previous reviews of the same restaurant, which can be explained by elite reviewers having a greater reputation concern on Yelp. These reputation concerns (i.e. popularity concerns), as well as the high signal precision of elite reviewers, suggest that the aggregate rating should give more weight to elite reviews. However, at least in our data, reviewer heterogeneity in signal precision and reputational concern explain much less of the optimal-vs-simple-average 1 Elite status initiates from a nomination from the Yelp reviewers (can be the reviewer herself), and the final designation decision is made by Yelp based on the reviewer’s Yelp activeness. 2 difference than the martingale evolution of restaurant quality and the overall time trend in consumer reviews. Although our algorithm is derived from Yelp reviews, it could be applied to virtually any website that relies on consumer ratings to convey information of product or service quality. This contributes to the small, but growing literature on information aggregation as well as the literature on consumer reviews. Li and Hitt (2008) find that book reviews on Amazon tend to trend downward overtime, which they attribute to selection, with early purchasers tending to be those who have the strongest preferences for the book, providing further motivation for the need for optimal aggregation. Glazer et al. (2008) have theoretically considered optimal ratings in the context of health plan report cards. Another approach to aggregate the information is via demand estimation. Based on hotel reservation data from Travelocity.com, which include consumer-generated reviews from Travelocity.com and TripAdvisor.com, Ghose, Ipeirotis and Li (forthcoming) estimate consumer demand for various product attributes and then rank products according to estimated “expected utility gain.” In comparison, we attempt to aggregate consumer reviews without complementary data on how consumers use such reviews when they choose a product. This situation is faced by many opinion generation websites that offer consumer ratings but do not sell the rated products. Readers interested in consumer usage of Yelp reviews can refer to Luca (2011) who combines the same Yelp data as in this paper with restaurant revenue data from Seattle2 . Finally, our model of reputation concerns is also related to the vast literature on information cascade and the growing literature on observation learning (e.g. Banerjee 1993). The rest of the paper is organized as follows. Section 2 presents the model and describes how we estimate and identify key parameters in the model. Section 3 describes the data and presents reduced-form results. Section 4 presents structural estimates. Section 5 presents counterfactual simulations, and compares optimal average ratings to arithmetic average ratings. Section 6 concludes. 2 Model and Estimation Consider a consumer review website that has already gathered many consumer reviews on many products over a period of time. Our goal is to optimally summarize existing reviews into a single metric of concurrent quality for each product. Simple average assumes that every consumer review follows an i.i.d. distribution around a stable level of true product quality. This assumption can be violated if true quality evolves over time, if reviews are sequentially 2 More generally, there is strong evidence that consumer reviews are an important source of information in a variety of settings. Chevalier and Mayzlin (2006) find predictive power of consumer rating on book sales. Both Godes and Mayzlin (2004) and Duan, Gu, and Whinston (2008) find the spread of word-of-mouth affect sales by bringing the consumer awareness of consumers, the former measure the spread by the “the dispersion of conversations across communities” and the latter by the volume of reviews. Duan et al. (2008) argues that after the endogenous correlation among ratings, online user reviews has no significant impact on movies’ box o⇥ce revenues. 3 correlated, and if reviewers differ in stringency and accuracy. This section presents a structural model that captures all these elements in a coherent framework. Our goal in the model is to incorporate economically important parameters while maintaining econometric tractability. 2.1 Basic Setup Consider reviewer i who writes a review for restaurant r at calendar time tn .3 As the nth reviewer of r, she observes her own signal srtn as well as all the n {xr1 , xr2 , ..., xrn 1 }. 1 reviews of r before her srtn is assumed to be an unbiased but noisy signal of the true quality µrtn such that srtn = µrtn + ⌅rn where ⌅rn ⌃ N (0, ↵i2 ). We assume the noise has the same variance when reviewer i visits different restaurants. This way, we can denote the precision of reviewer i’s information as vi = 1 . ⇤i2 Because r and n jointly identify a unique reviewer, we use i interchangeably with the combination of r and n. We consider two incentives for reviewer i to report. The first incentive is to speak out her own emotion and obtain personal satisfaction from it. If satisfaction comes from expressing the true feeling, this incentive motivates her to report her own signal. If i obtains psychological gains from reporting the signal with certain deviation, which we denote as stringency ⌃rn = 0, then she will be motivated to report her signal plus her stringency measure.4 The second incentive of submitting a Yelp review is to make a best guess of restaurant quality so that she can earn popularity or reputation in the future. For simplicity, we assume popularity is defined by the extent to which Yelp readers agree with her review. If there are many more readers than reviewers and each reader expresses her true feeling when she collates her experience with the review, popularity of i would decrease with the distance between her review xrtn and the restaurant’s actual quality µrtn .5 Combining the above two incentives, reviewer i, as the nth reviewer of restaurant r, chooses her review xrtn in order to minimize the following objective function: Frn = (1 ⌦i )(xrtn srtn where E(µrtn |xrt1 , xrt2 , ...xrt{n 1} ⌃rn )2 + ⌦i [xrtn E(µrtn |xrt1 , xrt2 , ...xrtn 1 , srtn )]2 , srtn ) is the posterior belief of true quality µrtn and ⌦i is the weight that i puts on her popularity on Yelp. The optimal review to minimize Frn is: xrtn = (1 ⌦i )(⌃rn + srtn ) + ⌦i E(µrtn |xrt1 , xrt2 , ..xrtn 1 , srtn ) = ⌥rn + (1 where ⌥rn = (1 ⌦i )srtn + ⌦i E(µrtn |xrt1 , xrt2 , ..xrtn 1 , srtn ) ⌦i )⌃rn represents the stringency or bias of reviewer i for restaurant r. 3 We assume that a reviewer submits one review for a restaurant. Therefore, the order of the review indicates the reviewer’s identity. On Yelp.com, reviewers are only allowed to display one review per restaurant. 4 Some reviewers are by nature generous and obtain psychological gains from submitting reviews that are more favorable than what they actually feel. In this case, rn > 0 represents leniency. 5 In this sense the reviewer is a truth seeker. Another interpretation of the popularity incentive is the desire to contribute to a public good by submitting the best estimate of restaurant quality. 4 Note that popularity weight ⌦i can be negative if reviewer i enjoys being far away from the best estimate of restaurant quality. This incentive to deviate is different from stringency ⌃rn because ⌃rn is independent of any reviewer before i but the incentive to deviate due to negative ⌦i depends on the reviews available before tn . Popularity weight ⌦i allows sequential correlations among reviews of the same restaurant, even after taking out restaurant fixed effects. This is because a reviewer that cares about popularity on Yelp will extract useful information from past reviews and incorporate them into her own estimate of restaurant quality. This assumption is consistent with the findings of Chen et al. (2010), who use a field experiment on MovieLens to show that some users dramatically change their movie ratings when they are presented with some social information (e.g. the median user’s total number of movie ratings). Of course, this is not the only explanation for correlations across reviews; one can argue that the first review attracts certain types of patrons who have similar (or opposite) tastes to the first reviewer and this selection process generates correlations across reviews. Without external information to pin down such selection, the resulting sequence of reviews can be observationally equivalent to what arises from popularity concerns. In this sense, we would like readers to take popularity weight ⌦i as an indicator of how a review correlates with past reviews. As long as later reviews capture information from past reviews, optimal aggregation needs to weigh early and late reviews differently. 2.2 Restaurant Quality Change If restaurant quality is constant over time and every reviewer is unbiased, then aggregation of consumer reviews is straightforward: even a simple average of reviews will generate an unbiased indicator of true quality, and optimal aggregation can only improve e⇥ciency by giving more weight to more precise reviewers or reviewers with greater popularity concerns. However, the assumption of constant restaurant quality is unrealistic. The restaurant industry is known for high labor turnover as well as high entry and exit rates. A new chef or a new manager could change a restaurant significantly; even a sloppy waiter could generate massive consumer complaints in a short time. In reality, consumer reviews and restaurant quality may move together because reviews reflect restaurant quality, or restaurant owners may adjust a restaurant’s menu, management style, or labor force in response to consumer reviews. Without any direct data on restaurant quality, it is di⇥cult to separate the two. In light of the di⇥culty, we impose an independent structure on restaurant quality change and shy away from an endogenous generation of restaurant quality in response to consumer reviews. This way, we focus on measures of restaurant quality rather than reasons underlying quality change. In particular, we assume quality evolution follows a martingale process: µrt = µr(t 5 1) + t where t denotes the units of calendar time since restaurant r has first been reviewed and the t-specific evolution t conforms to t ⌃ i.i, d N (0, ↵⇥2 ). This martingale process introduces a positive correlation of restaurant quality over time, Cov(µrt , µrt⇤ ) = E(µr0 + t X ⇤ E(µrt ))(µr0 + ⌅ ⌅ =1 = E( t X ⌅ ⌅ =1 E(µrt⇤ )) ⌅ ⌅ =1 ⇤ t X t X ⌅) = ⌅ =1 t X E( ⌅2 ) if t < t⇤ , ⌅ =1 which increases with the timing of the earlier date (t) but is independent of the time between t and t⇤ . Recall that xrtn is the nth review written at time tn since r was first reviewed. We can express the nth reviewer’s signal as: srtn = µrtn + ⌅rn where µrtn = µrtn 1 + tn 1 +1 + tn 1 +2 + ... + tn . Signal noise ⌅rn is assumed to be i.i.d. with V ar(srtn |µrtn ) = ↵i2 where i is the identity of the nth reviewer. The variance of restaurant quality at tn conditional on quality at tn V ar(µrtn |µrtn 1 ) = V ar( tn 1 +1 + tn 1 +2 + ... + tn ) = (tn tn 2 1 )↵⇥ = 1 is, tn ↵⇥2 . Note that the martingale assumption entails two features in the stochastic process: first, conditional on µrtn 1 , µrtn is independent of the past signals {srt1 , ..., srtn 1 }; second, conditional on µrtn , srtn is independent of the past signals {srt1 , ..., srtn 1 }. As shown later, these two features greatly facilitate reviewer n’s Bayesian estimate of restaurant quality. This is also why we choose martingale over other statistical processes (such as AR(1)). 2.3 Reviewer Heterogeneity and Reviewer-Restaurant Match In addition to random changes of restaurant quality and random noise in reviewer signal, reviewers may differ in stringency, popularity concern, and signal precision. Optimal information aggregation - in our definition - would correct for these differences. For the purposes of this paper, we are estimating match based on characteristics that we observer - which is the review history for each reviewer and their Yelp-granted elite status. Yelp assigns elite status to a subset of reviewers who have been nominated - either by themselves or by other Yelp users - due to a perceived high quality of reviews. We take Yelp “elite” as a signal of a reviewer’s type, and hence take elite status as given. We then allow elite 2 }. If elite reviewers are reviewers to have {⌦e , ↵e2 } while all non-elite reviewers have {⌦ne , ↵ne able to obtain more precise signals of restaurant quality and care more about their reputation 2 . Elite and non-elite reviewers could also differ in on Yelp, we expect ⌦e > ⌦ne and ↵e2 < ↵ne 6 stringency ⌥e and ⌥ne . However, we do not know true restaurant quality and can at best only identify the stringency difference between elite and non-elite reviewers. In theory, reviewer stringency, popularity concern and signal precision can all vary over time. From observed reviewer history, we define several reviewer attributes at the time of a particular review. One is the number of reviews that reviewer i has submitted for Seattle restaurants before writing a new review for restaurant r at time t. This reflects reviewer experience with Seattle restaurants. We denote it as N umRevit . The second is review frequency of i at t, defined as the number of reviews i has submitted up to t divided by the number of calendar days from her first review to t. Review frequency allows us to capture the possibility that a reviewer who has submitted two reviews 10 months apart is fundamentally different from a reviewer who has submitted two reviews within two days, even though both reviewers have the same number of reviews on Yelp. We denote review frequency of i at t as F reqRevit . The third and fourth reviewer attributes attempt to capture reviewer-restaurant match. In reality, reviewers may have their own preference for cuisine type and sort themselves into different restaurants at different times. Although we do not have enough information to model the sorting explicitly, we can describe reviewer-restaurant match by characteristics of the restaurants a reviewer has written reviews for in the past. In particular, we collect 15 cuisine type indicators describing whether a restaurant is traditional American, new American, European, Mediterranean, Latin American, Asian, Japanese, seafood, fast food, lounge, bar, bakery/coffee, vegetarian, or others. These categories are defined by Yelp and not mutually exclusive. We also use Yelp’s definition of price categories (1,2,3,4) and code a missing price category as category 0. With these restaurant characteristics in hand, we use factor analysis to decompose them into eight orthogonal factors Fr = [fr,1 , ..., fr,8 ]. By construction, the sample mean of each factor is normalized to 0 and sample variance normalized to 1. We then collapse a reviewer history into two metrics: the first metric, Cit , measures the average restaurant that this reviewer has written reviews for before she writes her mth review at time t, the second metric, T asteV arit , measures the variety of restaurants that she has written reviews for before her mth review at time t. In particular, they are defined as: Cit = 1 m v u 8 uX T asteV arit = t q=1 1 m X1 1 m Fil , l=1 2 m X1 (fil,q f il,q )2 l=1 where m 1 is the number of Seattle restaurants reviewer i has written reviews for before t, Fil P 1 denotes the vector of factors of the lth restaurant that i visited, and f il,q = m1 1 m l=1 fil,q is the mean in factor q among the m 1 restaurants that i visited. If reviewer i has not reviewed 7 any restaurant yet, we set her taste equal to the mean characteristics of restaurants (Cit = 0). When reviewer i writes a review for restaurant r, we have a pair of {Cit , Fr } to describe the reviewer taste and restaurant characteristics. Assuming that reviewer i reviews restaurant r at time t, we define the reviewer-restaurant matching distance M atchDrit as M atchDrit = (Cit Fr )⇤ (Cit Fr ). The smaller the matching distance (M atchDrit ) , the better the match is between the restaurant and the reviewer’s review history. To summarize, we have five reviewer attributes: elite status (Elitei ), number of reviews (N umRevit ), frequency of reviews (F reqRevit ), matching distance between reviewer and restaurant (M atchDrit ), and taste for variety (T asteV arit ). By construction, all but Elitei vary within a reviewer over time, and only M atchDrit depends on the restaurant that the reviewer is about to review at time t. Readers should take M atchDrit and T asteV arit as controls for observable sorting between restaurants and reviewers. In reality, who reviews which restaurant at what time can be driven by past reviews of every restaurant and thus endogenous. Some unobservable tastes of reviewers will lead to specific values of M atchDrit and T asteV arit ; hence controlling for M atchDrit and T asteV arit indirectly controls for these unobservable tastes. Other unobservable attributes of reviewers may not have any influence on M atchDrit and T asteV arit , but they affect how reviewers read past reviews and then visit the restaurant and write their own reviews. This will generate correlations along the order of reviews, and such correlations are captured in popularity weight ⌦i . 2.4 Time Trend In addition to all the above, we also record the number of calendar days since restaurant r received its first review on Yelp until a reviewer is about to enter the review for r at time t. This variable, denoted as Agert , attempts to capture any linear trend in consumer reviews that is missed by the above-mentioned reviewer or restaurant variables. By definition, this trend – which turns out to be negative over time – is subject to multiple interpretations. It is possible that true restaurant quality declines over time for every restaurant. Note that this decline is in addition to the martingale evolution of restaurant quality because the martingale deviation is assumed to have mean zero. It is also possible that later reviewers are always harsher than early reviewers. Either interpretation can be a result of a “chilling” effect as described in Li and Hitt (2008), who also lay out these competing hypotheses. We are unable to distinguish these underlying stories, although the interpretation does affect how we calculate optimal estimate of true quality. We will come back to this point when we present the optimal average ratings in Section 5. To summarize, we assume: 8 ⌦i = Elitei · ⌦e + (1 ↵i2 = Elitei · ⌥ri = Agert · ↵e2 + (1 age + M atchDrit · + Elitei · [⌥(e Elitei ) · ⌦ne 2 Elitei ) · ↵ne + N umRevit · matchd ne)0 numrev + F reqRevit · + T asteV arit · f reqrev tastevar + Agert · ⇥day + N umRevit · ⇥numrev + F reqRevit · ⇥f reqrev + M atchDrit · ⇥matchd + T asteV arit · ⇥tastevar ] 2 } capwhere {⌦e , ⌦ne } capture the popularity concern of elite and non-elite reviewers, {↵e2 , ↵ne ture the signal precision of elite and non-elite reviewers, { quality or stringency change, { f reqrev , matchd , tastevar } age } captures the catch-all trend in capture how restaurant and reviewer attributes change the stringency of non-elite reviewers, and {⌥(e ne)0 , ⇥ day , ⇥numrev , ⇥f reqrev , ⇥matchd , ⇥tastevar }capture how restaurant and reviewer attribute change the stringency difference between elite and non-elite reviewers. We have tried to allow ⌦i and ↵i2 to vary by restaurant and reviewer attributes other than elite status, but none of them turns out to be significant from zero, so we ignore them here for the simplicity of illustration. 2.5 Data Generation Process The above model includes random change in restaurant quality, random noise in reviewer signal, reviewer heterogeneity in stringency, popularity concern, and signal precision, and a linear time trend, as well as the quality of the match between the reviewer and the restaurant. Overall, one can consider the data generation process as the following three steps: 1. Restaurant r starts with an initial quality µr0 when it is first reviewed on Yelp. Denote this time as time 0. Since time 0, restaurant quality µr evolves in a martingale process by calendar time, where an i.i.d. quality noise quality at t so that µrt = µr(t 1) + t. t ⌃ N (0, ↵⇥2 ) is added on to restaurant 2. A reviewer arrives at restaurant r at time tn as r’s nth reviewer. She observes the attributes and ratings of all the previous n 1 reviewers of r. She also obtains a signal srtn = µrtn + ⌅rn of the concurrent restaurant quality where the signal noise ⌅rn ⌃ N (0, ↵ 2 ) . 3. The reviewer chooses an optimal review that minimizes her loss of deviating from her own experience and her best estimate of concurrent restaurant quality. The optimal review takes the form xrtn = ⌥rn + ⌦n E(µrtn |xrt1 , xrt2 , .., xrt2 , ..., srtn ) + (1 9 ⌦n )srtn where E(µrtn |xrt1 , xrt2 , .., xrt2 , ..., srtn ) is the best guess of the restaurant quality at tn by Bayesian updating. 4. The reviewer is assumed to know the attributes of all past reviewers so that she can debias the stringency of past reviewers. The reviewer also knows that there is a linear trend in reviewer stringency which changes ⌥ by d⌥ per unit of calender time. If there is a linear trend in restaurant quality, it is just a linear term added to E(µrtn |xrt1 , .., xrtn , srtn ) and not distinguishable from the linear trend in ⌥ in the above expression for xrtn . In the Bayesian estimate of E(µrtn |xrt1 , xrt2 , .., xrt2 , ..., srtn ), we assume the nth reviewer of r is fully rational and has perfect information about the other reviewers’ observable attributes, which according to our model determines the other reviewers’ stringency (⌥), popularity preference (⌦), and signal noise (↵ ). With this knowledge, the nth reviewer of r can back out each reviewer’s signal before her thus the Bayesian estimate of E(µrtn |xrt1 , xrt2 , .., xrt2 , ..., srtn ) can be rewritten as E(µrtn |srt1 , ...srtn ). Typical Bayesian inference implies that reviewer’s posterior about restaurant quality is a weighted average of previous signals and her own signal, while weight increases with signal precision. What is complicated is that restaurant quality evolves by martingale and therefore current restaurant quality is better reflected in recent reviews. Accordingly, the Bayesian estimate of E(µrtn |srt1 , ...srtn ) should give more weight to more recent reviews even if all reviewers have the same stringency, popularity preference and signal precision. The analytical derivation of E(µrtn |srt1 , ...srtn ) is presented in Appendix A. 2.6 Maximum Likelihood Estimation According to the derivation of E(µrtn |srt1 , ...srtn ) in Appendix A, we can write out the probability distribution of all the Nr reviews of restaurant r, namely L(xrt1 , xrt2 , ..xrtNr ), and then estimate parameters by maximizing the combined log likelihood of all reviews of all R restauP rants logL = R r=1 logL(xrt1 , xrt2 , ..xrtNr ). The parameters to be estimated are restaurant quality at time 0 ({µr0 }R r=1 ), the standard deviation of the martingale noise of restaurant qual- ity change (↵⇥ ), the standard deviation of reviewer signal noise (↵e , ↵ne ), reviewer popularity concerns (⌦e , ⌦ne ), parameters affecting reviewer stringency ( , ⌥(e ne)0 trend ( numrev , f reqrev , matchd , tastevar , ⇥ day , ⇥numrev , ⇥f reqrev , ⇥matchd , ⇥tastevar ), and the parameter for the catch-all time age ). Note that consistent estimation of all other parameters depends on the consistency of {µr0 }R r=1 , which requires the number of reviews of each restaurant goes to infinity. But in our data, the number of reviews per restaurant has a mean of 33 and a median of 14. When we use simulated data to test the MLE estimation of observed reviews, we find that poor convergence of {µr0 }R r=1 affects the estimation of other key parameters of interest. To circumvent the problem, we estimate the joint likelihood of {xr2 xr1 , xr3 xr2 , ..., xrNr xrNr R 1 }r=1 instead. In this way we subtract the initial restaurant qualities {µr0 }R r=1 and only need to estimate the other parameters. Because the covariance structure of {xrt2 10 xrt1 , xrt3 xrt2 , ..., xrtNr xrtNr likelihood f (xrt2 f (xrt2 1 } is complicated, we use the change of variable technique to express the xrt1 , ..., xrtNr xrt1 , ..., xrtNr xrtNr xrtNr More specifically, f (xrt2 1 1 ) by f (srt2 ) = |J • Step 1: To derive f (srt2 1 s⇥ x | xrt1 , ..., xrtNr xrtNr srt1 , ..., srtNr srt1 , ..., srtNr 1 f (srt2 srtNr 1 srt1 , ..., srtNr ), srtNr 1 ). ) is calculated in three steps: srtNr 1 ), we note that srtn = µrtn + ⌅n and thus, for any m > n, n ⇧ 2, the variance and covariance structure can be written as: Cov(srtn srtn 1 , srtm srtm 1 ) =Cov(⌅rn ⌅rn 1 + tn 1 +1 + ... + 8 < ↵ 2 if m = n + 1 rn = :0 if m > n + 1 V ar(srtn 2 2 =↵rn + ↵rn tn , ⌅rm ⌅rm 1 + tm 1 +1 + ... + tm ) srtn 1 ) 1 + (tn tn 2 1 )↵⇥ . Denoting the total number of reviewers on restaurant r as Nr , the vector of the first differences of signals as as ⇥ sr , r srtn 1 }N n=2 , and its covariance variance structure sr = {srtn we have Nr 1 2 f ( sr ) = (2 ) |⇥ sr | (Nr 1)/2 exp( 1 ⇤ s ⇥ 2 r 1 sr sr ). R • Step 2: We derive the value of {srt , ...srtNr }R r=1 from observed ratings {xrt1 , ...xrtNr }r=1 . Given xrtn = ⌥rn + ⌦n E(µrtn |srt1 , ...srtn ) + (1 ⌦n )srtn and E(µrtn |srt , ...srtn ) as a function of {srt1 , ...srtn } (formula in Appendix A), we can solve {srt1 , ...srtn } from {xrt1 , ...xrtn } according to the recursive formula in Appendix B. • Step 3: We derive |J 2 6 4 s⇥ x | or |J 1 srt2 3 srt1 ... srtn the analytical form of J srtn x⇥ s x⇥ s |, 1 7 5=J where J 2 6 x⇥ s 4 x⇥ s xrt2 is such that 3 xrt1 ... xrtn xrtn 1 7 5 is available given the recursive expression for xrtn and srtn . 11 2.7 Identification Since our model includes restaurant fixed effects (denoted as time-0 quality µr0 ), all our parameters are identified from within-restaurant variations. In particular, reviewer popularity weight ⌦ and signal variance ↵ 2 are identified by the variance-covariance structure of reviews within a restaurant. To see this point, consider a simple case where restaurant quality is stable (i.e. ↵⇥2 = 0). If every one has the same signal variance ↵ 2 , for the nth review, we have V ar(xrn ) = ⌦n (2 ⌦n ) ↵2 + (1 n ⌦n )2 ↵ 2 . As we expect, it degenerates to ↵ 2 if the nth reviewer puts zero weight on popularity (⌦n = 0). When ⌦n > 0, V ar(xrn ) declines with n. If the nth reviewer cares about popularity only (⌦n = 1), we have the familiar form of V ar(xrn ) = ⇤2 n . In other words, ⌦ determines the degree to which the variance of reviews shrinks over time, while ↵ 2 determines the variance of the first review. There are overidentifications for ⌦ and ↵ 2 , because they affect not only the variance of reviews but also the covariance between reviews. In the above simple case, the covariance of xrm and xrn for m < n is: ⌦n Cov(xrm , xrn ) = Pn j=1 vj which declines with n, increases with ⌦n , and does not depend on the distance between m and n. This is because the covariance of reviews is generated from reviewer n’s belief of restaurant quality, and reviewer n values the information content of each review equally according to the Bayesian principle. Nevertheless, popularity concern is not the only force that generates correlation between reviews within a restaurant. The other force is restaurant quality evolution. How do we separate the two? The above description has considered the case with popularity concern but no restaurant quality change (↵⇥2 = 0 and ⌦ > 0). Now let us consider a model with ↵⇥2 > 0 and ⌦ = 0, which implies that restaurant quality evolves over time but reviewers do not incorporate information from previous reviews. In this case, the correlation between the nth and the (n k)th reviews only depends on the common quality evolution before the (n reviewer, not the order distance (k) or time distance (tn In the third case of ↵⇥2 > 0 and ⌦ > 0, the nth tn k) k)th between the two reviews. reviewer is aware of quality evolution and therefore put more weight on recent reviews and less weight on distant reviews. In particular, one can show that the correlation between the nth and the (n k)th reviews depends on not only the order of review but also the time distance between the two reviews. In short, the separate identification of the noise in quality evolution (↵⇥2 ) from reviewer popularity concern and signal precision{⌦, ↵ 2 } comes from the calendar time distance between reviews. 12 As stated before, we allow both ⌦ and ↵ 2 to differ between elite and non-elite reviewers. 2 } are identified by the varianceBecause we observe who is elite and who is not, {⌦e , ↵e2 , ⌦ne , ↵ne covariance structure of reviews as well as the arrival order of elite and non-elite reviewers. The constant bias difference between elite and non-elite reviewers ( ⌥(e ne)0 ) is identified by the mean difference of elite and non-elite reviews on the same restaurant. The other parameters that affect reviewer stringency, namely { age , numrev , f reqrev , matchd , tastevar }, {⇥ day , ⇥numrev , ⇥f reqrev , ⇥matchd , ⇥tastevar }, are identified by how the observed reviews vary by restaurant age, reviewer attributes at time t, reviewer-restaurant match, and their interaction with elite status. 2.8 Optimal Estimate of Restaurant Quality Following the above model, if we interpret the linear trend of ratings (Agert · age in ⌥rn ) as reviewer bias (relative to the first reviewer), the optimal estimate of restaurant quality at time tn is defined as E(µrtn |xrt1 , xrt2 , .., xrtn ), which is equivalent to E(µrtn |srt1 , srt2 , .., srtn ) and we know how to calculate it according to Appendix A. If we interpret the linear trend of ratings (Agert · age in ⌥rn ) as changes in true quality, the optimal estimate of quality at tn is E(µrtn |xrt1 , xrt2 , .., xrtn ) + Agertn · 3 age . We will report both in Section 6. Data and Reduced Form Results Our empirical setting is the consumer review website Yelp.com. Yelp began in 2004, and contains reviews for a variety of services ranging from restaurants to barbers to dentists, among many others, although most Yelp reviews are for restaurants. For a more complete description of Yelp, see Luca (2011). In this paper, we use the complete set of restaurant reviews that Yelp displayed for Seattle, WA at our data download time in February 2010. In total, we observe 134,730 reviews for 4,101 Seattle restaurants in a 64-month period from October 15, 2004 to February 7, 20106 . These reviews come from 18,778 unique reviewers, of which 1,788 are elite reviewers and 16,990 are non-elite as of the end of our data period. Elite reviewers are determined via a nomination process, where a reviewer can self-nominate or be nominated by someone else. We do not observe the nomination process, and instead only observed whether someone ultimately becomes elite. For our purposes, we take elite status as fixed. Since Yelp reviewers can leave reviews for restaurants throughout the US but our data cover Seattle only, we do not have the complete Yelp history of each reviewer. Another data limitation is that our data contain star ratings given in each review (one to five), but do not include the text. Each reviewer is only allowed to display one review per restaurant, but Yelp allows reviewers to update their existing reviews. If a review has been updated, the review 6 Yelp identifies reviews that either violate terms of service or seem to be fake, as determined by an algorithm, and removes these reviews from the main Yelp webpage and ratings. We do not observe these reviews, and do not consider them in our analysis. 13 date records the time of update and there is no information indicating the date or content of the replaced review. Due to this data limit, we treat updated reviews the same as other reviews. In our data set, 64.53% of reviewers have written at least two reviews and 23.7% have written at least five reviews. which provide us with within-reviewer variation. Table 1 summarizes the main variables in our analysis. In the first panel of restaurant characteristics, we note that on average each restaurant receives 33 reviews but the distribution is highly skewed to the right, ranging from 1 to 698 with a standard deviation of 50 and median of 14. At the end of our data, an average restaurant receives 0.156 reviews per day. This masks enormous heterogeneity of review frequency; if we calculate review frequency per restaurant at any time of a new review, it varies from 0.001 to as large as 28 reviews per day. The arrival of reviews also varies over the lifetime of a restaurant: on average, the second review of a restaurant comes 155 days later than the first review, while the average lag is 34 days between the 11st and 12nd reviews and 21 days between the 21st and 22nd reviews. This is partly driven by the fact that most restaurants receive only a handful number of reviews far apart, while a small fraction of restaurants receive more reviews that arrive much more frequently. The second panel of Table 1 summarizes the data by reviewers. Although less than 10% of reviewers are elite, an average elite reviewer writes five times more reviews than a non-elite reviewer (24 versus 5). As a result, elite reviewers account for 32.5% of all reviews. Comparing elite and non-elite reviewers, they are similar in average rating per review (both around 3.7 stars), but elite reviewers have a higher review frequency, a closer match with the restaurants they review, and slightly higher variety of taste. The latter two are partly driven by elite reviewers writing more reviews in our data. 3.1 What Explains the Variation in Yelp ratings? Although the goal of Yelp is to provide information about a restaurant’s quality, there are many other factors that determine a restaurant’s Yelp rating, for all of the reasons discussed throughout this paper. To get a feel for how significant these other factors are, Table 2 presents the variance explained by different factors. A linear regression using reviewer fixed effects shows that reviewer fixed effects alone accounting for 23.3% of the total variation in Yelp ratings. This suggests that individual stringency can have a large effect on the final rating. One way to think about restaurant quality is to use restaurant fixed effects, and its variation alone explains 20.86% of total variation in Yelp ratings. Incorporating both reviewer and restaurant fixed effects, we can explain almost 36% of total variations. This is less than adding the variations accountable by reviewer or restaurant fixed effects separately, suggesting that there is systematic match between reviewers and restaurants. In fact, we are able to control for some of this matching through our proxies for match quality, which further explains the variation in Yelp ratings. 14 3.2 Elite Reviewers The data shows quite clearly that different reviewers behave differently. It is possible to segment these individuals into groups of reviewers. In particular, crowdsourced settings such as Yelp, TripAdvisor, and Wikipedia identify reviewers they expect to be especially influential and give them a special certification. On Yelp, this is the “elite” system. In this section, we investigate the ways in which elite reviewers differ from other reviewers. We are interested in this for two reasons. First, this is an increasingly common way to structure review websites, and therefore is of direct interest. Second, by segmenting reviewers, we can allow the weights assigned to a given review to endogenously adjust for different groups of reviewers. Table 3 shows that elite and non-elite reviews are, in fact, systematically different. The significantly negative coe⇥cient of the elite dummy suggests that elite reviews deviate less from the long-run average rating of the restaurant, suggesting that elite reviewers have more 2 ) or care more about their popularity on Yelp (⌦ > ⌦ ). precise signals (↵e2 < ↵ne e ne The elite versus non-elite difference of rating is also presented in Figure 1. The left (right) graph of Figure 1 shows the kernel density of a rating minus the restaurant’s average rating beforehand (afterward), for elite and non-elite reviewers separately. An elite reviewer tends to give a rating closer to the restaurant’s average ratings before or after her, one phenomenon to be expected if elite reviewers have either more precise signal or greater popularity concerns. 3.3 Dynamics of the Review Process This makes five empirical observations related to the dynamics of the reviews, which we use to inform our model. In this section, we describe these effects, and highlight the relationship between the reduced form and structural results. 3.3.1 Ratings are less variable over time As detailed in Section 2, identification of our model relies on the extent to which the variance of reviews shrinks over time within a restaurant. If reviewers tend to incorporate a restaurant’s previous reviews (⌦ > 0), we should observe reviews to vary less and less over time around the restaurant’s fixed effect. To check this intuition, we first obtain residual ⌅[ ri,yr after regressing observed rating on reviewer, restaurant and year fixed effects (i.e. xri = µr + i +⇤year +⌅ri,yr ), 2 and then associate residual square ([ ⌅ri,yr ) with the order of a review (Nri ) and whether the 2 review is written by an elite reviewer (i.e. ⌅[ ri,yr = ⇥0 + ⇥1 Dri,elite + ⇥2 Nri + ⇥3 Nri ⇥ Dri,elite + ⇧ri ). Table 3 shows the results of the latter regression. The significantly negative coe⇥cient on the order of review in suggests that reviews deviate less and less over time from the average rating of the restaurant. Such variance reduction is consistent with reviewers incorporating previous reviews in their own review (⌦ > 0). 15 3.3.2 Ratings trend downward over time Our data shows that ratings within a restaurant tend to decline over time. Figure 2 plots ⌅[ ri,yr by the order of reviews within a restaurant, in the fitted fractional polynomial smooth and the corresponding 95% confidence interval. This is consistent with Li and Hitt (2008), who document a downward trend in a product’s Amazon reviews over time. There are multiple factors that could contribute to this downward trend. First, it could be a selection effect, where a restaurant with a good rating tends to attract new customers who do not like the restaurant as much as the old clientele. This is the mechanism outlined by Li and Hitt (2008). If this were the primary driver of the result in our setting, then we would expect later reviewers to be a worse fit for the restaurant. We do find this, as discussed in section 3.3.4. However, this only explains a small component of the downward trend. Because any estimated matching index clearly does not capture all aspects of the match between a customer and a restaurant, we believe that there are contributing factors as well. An alternative possibility is that restaurants decline in quality over time. 3.3.3 Ratings are serially correlated Positive popularity concerns also imply positive serial correlation of ratings within a restaurant. In other words, we should see a stronger correlation between the 3rd and 4th review than between the 2nd and 4th. To check this, we regress the above-obtained rating residual ⌅[ ri,yr on its lags within the same restaurant. As shown in Table 4, the residuals show strong, positive correlation over time, while the correlation dampens gradually by the order distance between reviews. This is clear evidence that reviews cannot be treated i.i.d. as the simple-average aggregation assumes. 3.3.4 Restaurants find reviewers with less diverse taste over time Table 5 presents reduced-form analysis regarding variations of reviewer-restaurant matching distance (M atchDrit ) and reviewer’s taste for variety (T asteV arrit ). The first two columns of Table 5 show that, within a restaurant, later reviewers tend to have less of a diverse taste. This is consistent with the positive sorting hypothesis. And we did not find significant evidence for “chilling” effect where the matching between between restaurant and reviewers gets worse over time. 3.3.5 Reviewers find better matches over time The last two columns of Table 5 examine variations within a reviewer, which turn out to be quite different from what we have seen within a restaurant. Within a reviewer, the later visited (and reviewed) restaurants are better matched with the reviewer’s taste and the reviewer has more taste for variety when she visits and reviews the later restaurants. This suggests that an 16 average reviewer finds better matches over time, but is also more willing to seek variety. In other words, M atchDrit and T asteV arrit capture at least part of the dynamic sorting between restaurants and reviewers, although we do not model the sorting explicitly. 4 Results from Structural Estimation The goal of our model is to estimate parameters that can then be used for optimal information aggregation. As described in the model section, the parameters of interest pertain to (1) a reviewer’s stringency and accuracy, (2) the extent to which a reviewer takes into account prior reviews, (3) the likelihood that a restaurant has changed quality, and (4) the quality of the match between the reviewer and the restaurant. We allow these parameters to vary between groups of reviewers. As an example of how this would work, we compare the parameters for elite and non-elite reviewers. We choose these subsets of reviewers because elite reviewers are such a central part of the review system, as documented in section 3. Table 6 presents structural estimation results in four columns. In Column (1), we estimate the model under the assumptions that restaurant quality is fixed and reviewers have the same signal precision, popularity weight, and stringency.The estimated signal precision and popularity weight will ultimately be used to optimally aggregate reviews. Note that popularity weight is statistically different from zero, suggesting that reviewers are taking into account the content of previous reviews. As we will see in the simulation section, this will cause later reviews to receive more weight than early reviews. In the rest of this section, we relax the assumptions to allow for elite reviewers to have different reviewing behavior, and to allow restaurants to change quality over time. 4.1 Elite Reviewers In Column (2), we allow signal precision, popularity weight, and stringency to differ by reviewer type. Specifically, we allow for elite reviewers to behave differently than other reviewers. The estimates, as well as a likelihood ratio test between Columns (1) and (2), clearly suggests that elite and non-elite reviewers differ in both signal precision and popularity weight. Elite reviewers put higher weight on past reviews and have better signal precision. That being said, all reviewers put more than 75% weight on their own signals, and the noise in their signal is quite large considering the fact that the standard deviation of ratings in the whole sample is of similar magnitude as the estimated ↵e and ↵ne . In terms of stringency, Column (2) suggests insignificant difference between elite and non-elite reviewers. 4.2 Restaurants with Changing Quality Column (3) allows restaurant quality to change in a martingale process every quarter. As we expect, adding quality change absorbs part of the correlation across reviews, and has significantly reduced the estimate of ⌦, but the magnitude of ⌦e 17 ⌦ne is stable at roughly 11-12%. With quality change, ⌦ne is estimated to be significantly negative, suggesting that a non-elite reviewer tends to deviate from the mean perspective of the crowd before him, after we allow positive autocorrelation across reviews due to restaurant quality change. Compared to non-elite, elite reviewers are more positively influenced by the past crowd because their popularity/reputation concerns motivate them to be closer to the other reviewers. Although the quarterly noise in restaurant quality (↵⇥ = 0.1452) is estimated much smaller than the noise in reviewer signal (↵e = 0.9293 and ↵ne = 0.9850), this amounts to substantial noise over the whole data period because a random draw of adds up to restaurant quality every quarter. A likelihood ratio test between Column 3 and Column 2 favors the inclusion of restaurant quality change. In addition to restaurant quality change, Column (4) allows reviewer stringency to vary by: • the restaurant’s tenure (on Yelp) Agert , the quality of the match between the reviewer and the restaurant M atchDrit , • the reviewer’s taste for variety T asteV arri , • the number of reviews a reviewer has written N umRevit , • the frequency with which a reviewer writes reviews RevF reqit , and • the reviewer’s elite status. The set of coe⇥cients that starts with µ + ⌥ne describes the stringency of non-elite reviewers (which are not identifiable from the time-0 restaurant quality), while the set of coe⇥cients that starts with ⌥e ⌥ne describes the stringency difference between elite and non-elite reviewers. According to these coe⇥cients, reviewers are more stringent over time, indicating that there is a “chilling effect.” This “chilling” effect is less for elite reviewers. Moreover, reviewers that have written more reviews on Yelp match better with a restaurant, and have more diverse tastes tend to be more stringent. In comparison, an elite reviewer behaves similar in terms of matching distance and taste for variety, but her stringency does not vary significantly by the number of reviews on Yelp. Again, likelihood ratio tests favor the full model of Column 4 over Columns 1-3, suggesting that it is important to incorporate restaurant quality change, reviewer heterogeneity, and signal noise all at once. A remaining question is, at what frequency does restaurant quality evolve? Given the lack of hard evidence on this, Table 7 shows the full model estimation results with restaurant quality evolving by month, quarter, and half-year. The main changes occur in the estimates for noise of reviewer signal (↵e , ↵ne ), noise of quality evolution (↵⇥ ), and reviewers’ popularity weight (⌦e , ⌦ne ). This is not surprising because they are all identified by the variance-covariance structure of reviews within a restaurant. Nevertheless, we are able to identify quality evolution from reviewer signal and popularity preference because there are enormous variations in how 18 close sequential reviews arrive. Clearly, the more frequent we allow restaurant quality to vary, the smaller ↵⇥ is (because it captures quality change in a smaller calendar window), the lower the popularity weight, and and the lower the reviewers’ signal noise (because more frequent quality change absorbs more autocorrelations of nearby reviews). However, the difference between elite and non-elite reviewers remains similar across the three columns of Table 7. The likelihood reported at the end of Table 7 suggests that the raw data are better explained by more frequent changes of restaurant quality. 4.3 Comparing to a Model of Limited Attention One assumption underlying our structural model is reviewer rationality. One may argue that the assumption of full rationality is unrealistic, given consumer preference for simple and easyto-understand metrics. Anecdotally, we know reviewers tend to pay more attention to detailed information than those who only read reviews. To address the concern more rigorously, we estimate an alternative model in which we assume that reviewers are naive and use the simple average of a restaurant’s past rating as the best guess of quality. Recall in the full model that the nth reviewer’s optimal review should be xrtn = (1 ⌦n )(⌃rn + srtn ) + ⌦n E(µrtn |xrt1 , xrt2 , ..xrtn 1 , srtn ) where E(µrtn |xrt1 , xrt2 , ...xrtn 1 , srtn ) is the Bayesian posterior belief of true quality µrtn . If the reviewer is naive, the optimal review will change to: xrtn = (1 ⌦n ) ⇥ (⌃rn + srtn ) + ⌦n ⇥ ( where a simple average of past reviews estimate of quality E(µrtn |xrt1 , xrt2 , ..xrt{n 1} 1 n 1 Pn 1 n 1 i=1 xrti 1 n X1 xrti ) i=1 replaces the Bayesian posterior , srtn ). In an unreported table, we compare the MLE result and log the likelihood of the Bayesian and naive models, while allowing restaurant quality to update by quarter or half year.7 According to the Akaike information criterion (Akaike 1974), if we assume quality updates by half year, the Bayesian model is 49,020.8 times as probable as the naive model to minimize the information loss.8 Similarly, if we assume quality updates by quarter, we find the Bayesian model is 2.41 ⇥ 108 times as probable as the naive model. This suggests that the Bayesian model is more suitable for our data. 7 In all specifications, we assume that the reviewer stringency term (⇥rt ) only depends on M atchDrit , T asteV arit , and Agert . Later on, we will redo this by adding N umRev and F reqRev in ⇥rt but they are unlikely to change the results. 8 Specifically, we have exp(AICBayesian AICN aive ) = exp(logLBayesian logLN aive ) = 49, 020.8. 19 5 Counterfactual Simulations This section presents three sets of counterfactual simulations. The first set highlights the role of each modeling element in the optimal aggregation of simulated ratings. The second set compares optimally aggregated ratings - as determined from our algorithm - to the arithmetic average ratings currently presented on Yelp (and many other websites). The third set shows the impact of shocks on a restaurant’s simple and optimal average ratings over time. 5.1 Counterfactuals Across Model Variations Based on Simulated Ratings The structural results presented in Section 4 stress the importance of incorporating many modeling elements in one single model. But how important is each element in its contribution to an optimal aggregation of ratings? When optimally aggregating, we are making adjustments to remove biases and choosing weights to maximize e⇥ciency. Essentially, we are trying to find weights that will lead to information that is unbiased and as precise as possible. We analyze this question through a series of counterfactual simulations. The condition in which the simple average is an unbiased and e⇥cient summary of restaurant quality is the following: reviewer signals are i.i.d., restaurant quality is stable, and there is no reviewer popularity weight or bias. To start, we take this condition as the benchmark. In order to highlight how much optimal average is superior than the simple average in each model variation, we add each variation separately to the benchmark and compare by simulation. For figures 3 and 4, we consider a hypothetical restaurant with a fixed true quality/rating. We then simulate the 95% confidence interval of average ratings that would occur under different aggregation procedures. For figure 5, we consider a hypothetical restaurant with quality change following a martingale process. Because the quality is random variable in this model, we compare the mean absolute error and the mean squared error of the two aggregation procedures in estimating the true restaurant quality when each review is written. The first model variation we consider allows reviewers to put positive weight on popularity. When popularity concern is the only deviation from the assumption that reviews are i.i.d., then the arithmetic average is unbiased but ine⇥cient. Because later reviews have already incorporated past reviews, an arithmetic average across all reviews assigns too much weight to early reviews. As a result, the optimal average of ratings should give more weight to later reviews. Figure 3 presents two cases, one with ⌦ = 1 and the other with ⌦ = 0.6, while restaurant quality is fixed at 3 and reviewer’s signal noise is fixed at ↵ = 1. We create these figures by simulating a large number of ratings according to the underlying model, and then computing optimal versus simple average of ratings at each time of review. As shown in Figure 3, optimal average is more e⇥cient than simple average, and the e⇥ciency improvement is greater if reviewers are more concerned about their Yelp popularity. However, the right graph suggests that e⇥ciency gain over simple average is small even if the popularity weight is as large as ⌦ = 0.6. Recall that our structural estimation of ⌦ never exceeds 0.25 , which suggests 20 that the e⇥ciency gain from accounting for ⌦ in the optimal average is likely small in the real data. The second model variation is to allow elite reviewers to have signals that are of different precision than non-elite reviewers. Again, since we are not allowing for reviewers to differ in stringency or for restaurants to change quality, an arithmetic average is going to be unbiased but ine⇥cient. Optimal aggregation endogenously assigns more weight to elite reviews, since elite reviewers have reviews that are more precise. As shown in Figure 4, the more precise the elite reviewers’ signals are relative to other reviewers, the larger the e⇥ciency gain is for optimal average versus simple average. The third model variation adds restaurant quality evolution to the benchmark. Unlike the first two deviations from an i.i.d. distribution of reviews, failing to account for quality change does lead to bias in the arithmetic average ratings. We present three graphs in Figure 5: the first two allow the variance to change every quarterly restaurant quality change with different standard deviation in the noise of quality update, while the third one allows restaurant quality to update monthly with the same ↵⇥ as in the second graph. Review frequency is simulated using its empirical distribution as in the raw data. Comparison across the three graphs suggests that the optimal average rating, which accounts for restaurant quality evolution, leads to significant reduction in mean square errors especially when quality update is noisy or frequent. All three graphs of Figure 5 show the advantage of optimal average over simple average as an estimator of the true restaurant quality when quality follows the assumed random process with initial quality 3. To illustrate the magnitude of bias of optimal and simple average in one realized path of quality, Figure 6 focuses on a hypothetical change of quality from 3 at the beginning, to 2.5 at the 20th review, and to 3.25 at the 40th review. Reviewers believe that true quality is updated by quarter. To focus on the effect of restaurant quality evolution, Figure 6 fixes review frequency at 4.5 days per review. As shown in Figure 6, optimal average tracks the actual quality change better than simple average. Figure 7 highlights the importance of reviewer stringency (and its heterogeneity). Compared to the benchmark situation, we allow reviewer stringency (⌥) to vary by restaurant and reviewer characteristics (including the time trend by restaurant age) according to the coe⇥cients presented in the last column of Table 6. Reviewer and restaurant characteristics are simulated using their empirical distribution as observed in the raw data. The first graph of Figure 7 assumes that the reviewer bias changes with restaurant age, but the restaurant quality does not. And the second graph of Figure 7 assumes that the reviewer bias does not change with restaurant age, and only the restaurant quality does. Both graphs show that optimal average has corrected the bias in reviewer stringency and therefore reflects the true quality, but simple average is biased due to the failure to correct reviewer bias. Figure 8 allows everything else (popularity weight, reviewer signal noise, and restaurant quality change by quarter) and reruns the simulation. Again, because the restaurant quality is random, we 21 use the mean absolute and squared errors to compare the optimal and simple average. The graphs shows that the optimal rating represents the restaurant quality better and the gain is larger when a restaurant receives larger number of reviews.9 5.2 Optimal Versus Simple Average for Real Yelp Data We now compare optimal and simple average based on real Yelp ratings as observed in our data. According to our structural estimates in Table 7, the noise of quality update (↵⇥ ) has a standard deviation of 0.081 per month, which amounts to an average deviation of 0.28 stars per year. This is a substantial variation over time as compared to the standard deviation of 1.14 stars in the whole data set over six years. Noise in reviewer signal is even larger, with a standard deviation estimated to be between 0.9 and 1. These two types of noise have different implications for the relative advantage of optimal average ratings: quality update implies that optimal average needs to give more weight to recent reviews, which is not taken into account by simple average rating. In comparison, simple average reduces the amount of signal noise by law of large number and will do so e⇥ciently unless different reviewers differ in signal precision. Our estimates show a relatively small difference between ↵e and ↵ne (⌅0.05), implying that optimal weighting due to reviewer heterogeneity in signal noise is unlikely to lead to large e⇥ciency improvement. Another difference between elite and non-elite reviewers is their weight on popularity, but the absolute magnitudes of ⌦e and ⌦ne never exceed 0.2, suggesting that the e⇥ciency gain of optimal average due to popularity concern is likely to be small as well. Reviewer’s stringency bias is important in magnitude. We know from Table 1 that on average the second review is 155 days apart from the first review; according to the coe⇥cients on Agert , the second reviewer (if non-elite) will give a rating 0.05 stars lower. Over the six year period in our data, the stringency difference could be as substantial as 0.66 stars. Including all these elements, we compute simple and optimal average ratings at the time of every observed rating. This calculation is done for monthly, quarterly, and half-year quality update separately, according to the structural estimates in Table 7. We then calculate the difference between simple and optimal average, µsimple rn µoptimal , for every observation, and rn plot the mean and confidence interval of this difference by the order of review. As shown in the first row of Figure 9 (assuming quality updates by quarter), simple average rating is on average close to optimal average, but the confidence interval of their difference ranges from -0.1 to 0.2 stars in early reviews and enlarges gradually as more reviews accumulate. Within each restaurant, we calculate the percent of observations in which simple average rating is more than 0.15 stars away from the optimal average rating. The bar chart on the right hand side plots the histogram of restaurants by this percent. For example, the 9 In the simulation with full model specifications, the assumption for restaurant age affecting restaurant quality or reviewer bias is nonessential for comparing the mean absolute and squared errors of the two aggregating methods. Optimal rating always corrects any bias in reviewer bias, and simple rating always reflects the sum of the changes in quality and reviewer bias. 22 second bar shows that roughly 124 restaurants (out of the total 4,101) have 5-10% of times with simple average ratings more than 0.15 stars away from the optimal. Overall, 1,630 restaurants have at least 30% of times with simple average ratings more than 0.15 stars away from the optimal. This suggests that optimal average rating is likely to generate substantial improvement over simple average. The middle and bottom row-blocks of Figure 9 lead to similar conclusion when we model quality update as monthly or half-year instead of quarterly. Table 8 attempts to decompose the forces driving the difference between optimal and simple average ratings. The above discussion treats time trend (Agert · age ) as reviewer stringency (relative to the first reviewer). By this setting, the optimal average, calculated by E(µrtn |xrt1 , xrt2 , .., xrtn ), corrects for this trend while the simple average does not. The first panel of Table 8 shows that, by this calculation, 53-55% of restaurants have simple average ratings more than 0.15 stars away from the optimal by the end of the sample, and 26-29% of restaurants have simple average more than 0.25 stars away from the optimal. If instead we interpret the time trend (Agert · age ) as changes in restaurant quality, the optimal average should be E(µrtn |xrt1 , xrt2 , .., xrtn ) + Agert · age . The second panel of Table 8 shows that, after counting the linear trend as quality change (which means it does not contribute to the difference between optimal and simple average), the absolute difference between simple and optimal average rating is still more than 0.15 stars for 25-27% of restaurants, and more than 0.25 stars for 8-10% of restaurants by the end of the data sample. Overall, in the Yelp setting, the difference between optimal and simple average is mostly driven by restaurant quality updates (↵⇥ ) and the time trend (Agert · age ), and less by popularity concerns (⌦), reviewer’s signal noise (↵ ), or other terms in reviewer stringency (⌥rt ). 5.3 Impact of Shocks on Simple and Optimal Averages Our last set of counterfactual simulations highlight the impact of shocks on rating summary. Following the previous example where a restaurant’s true quality starts at 3 stars, jumps down to 2.5 at the time of the 20th review, and jumps back to 3.25 at the 40th review, we consider two types of shocks: one is a shock on true restaurant quality, and the other is a shock in the form of an outlier rating from a particular reviewer. For the former, Figure 10 presents three cases where the true quality shock (at 1.5 stars) occurs exactly once at either the first, third, or fifth review. For each case, we simulate a large number of reviewer ratings and compute the mean of simple average and optimal average at each review time. The left graph of Figure 10 shows that, when the true quality shock occurs at the time of the first review, both simple average and optimal average captures this shock immediately. But over time when the shock is gone, optimal average recovers faster than simple average and better tracks the concurrent restaurant quality. This is because, in an environment of quarterly quality update, optimal average is designed to give more weight to 23 recent reviews. The other two graphs of Figure 10 confirm this understanding: when the true quality shock happens in the middle of the review history, optimal average is not only closer to the true quality right after the shock, but also catches up faster with the recovered quality later on. What if the shock occurs in a submitted review instead of true restaurant quality? We simulate this situation in Figure 11. In particular, the three graphs of Figure 11 assume an outlier review of 1.5 stars was submitted as the first, third, or fifth review separately, while the true restaurant quality remains at 3 stars until the time of the 20th review. All the other reviews are simulated in a large number according to the underlying model. We then plot true quality, simple average, and optimal average by order of review. The left graph of Figure 11 shows that, if the outlier review is the first review, over time optimal average has a better ability to shed the influence of this outlier review, because it gives more weight to recent reviews. The right and bottom graphs of Figure 11 suggest that optimal average is not always the best; because it gives more weight to recent reviews, it gives more weight to the outlier review right after it has been submitted, which makes optimal average ratings further away from the actual quality in the short window after the outlier review. However, for the same reason, optimal average also forgets about the outlier review faster than simple average, and better reflects true quality afterward. 6 Conclusion As consumer reviews are beginning to offer unprecedented amounts of information, this paper argues that the way in which information is aggregated becomes a central design question. To address this question, we offer a method to aggregate consumer ratings into an adjusted weighted average for a given product, where the weights and adjustments are based on the informational content of each review. The informational content, in turn, is empirically determined based on variation in the reviewer characteristics (and review histories), as well as the inferred likelihood that product quality has changed, with parameters set by a model of reviewer behavior. Using the model, it is clear that optimally aggregated information deviates significantly from arithmetic averages. Our method is applicable in a variety of settings. While we have focused on consumer reviews, we could also use this to aggregate expert opinion. In this section, we discuss limits of our model and directions for future research. 6.1 Selection of Reviewers One limitation of our paper is that we do not observe the selection of consumers who decide to leave a review. In practice, reviewers have selected to purchase a product and also selected to leave a review. In principle, selection into a product would tend to skew ratings upward (you are more likely to eat at a restaurant that you think is a good match). The decision to review has an ambiguous effect, depending on whether people are more likely to review something 24 after a good or bad experience. One could structurally measure this selection function by imposing further assumptions on restaurant preferences. In fact, the information systems literature has documented bimodal distributions of reviews in Amazon (Hu et al 2009), and attributed this to tendency to review when opinions are extreme. While we do not model this selection, we estimate the quality of a reviewer’s match to a restaurant using the history of reviews (e.g. some reviewers tend to leave more favorable reviews for Thai food, while others leave better reviews for pizza). Moreover, Yelp reviews do not have the bimodal distribution found by Hu et al as evidence of significant selection problems. This may in part be due to Yelp’s focus on encouraging social image and community interaction. We also account for time trend and serial correlation of Yelp reviews within a restaurant, both of which could be generated by reviewer selection. Two concurrent papers are currently investigating the selection process. Chan et al. (2010) uses a Bayesian learning model on data from a Chinese restaurant review website similar to Yelp.com in order to estimate the way consumers use reviews. They focus on studying how reviewer sorting is affected by social network connection and review content, so they do not consider reviewers’ strategic reporting behavior as well as quality change. Our objective is to uncover the optimal representation of restaurant quality, which is quite different from theirs. Wang et al. (2012) also study restaurant reviews, and examine how reviews can influence reviewer behavior in exploring new restaurant choices. Although consumers’ variety seeking behavior is not the main theme of our study, we treat it as a heterogeneous reviewer characteristic that may influence reviewer rating. We find that reviewers with a wider variety of reviewing experience are relatively more stringent in ratings. 6.2 Incentives to Write Reviews Our paper has focused on taking an existing set of reviews and optimally aggregating them to best reflect the quality of a product. An alternative mechanism to achieve this goal is to use incentives to encourage people to leave more representative reviews. These incentives often seem to rely on social image. There is a large theoretical literature studying social image (Akerlof 1980, Bénabou and Tirole 2006). Theoretically modelling a crowdsourced setting, Miller, Resnick and Zeckhauser (2005) present a model arguing that an effective way to encourage high-quality review is rewarding reviewers if their ratings predict peer ratings. Consistent with this theory, Yelp allows members to evaluate each other’s reviews, chat online, follow particular reviewers, and meet at o⇤ine social events. It also awards elite status to some qualified reviewers who have written a large number of reviews on Yelp. As shown in our estimation, elite reviewers are indeed more consistent with peer ratings, have more precise signals, and place more weight on past reviews of the same restaurant. Wang (2010) compares Yelp reviewers with reviewers on completely anonymous websites such as CitySearch and Yahoo Local. He finds that Yelp reviewers are more likely to write more reviews, productive reviewers are less likely to give extreme ratings, and the same restaurants are less likely to 25 receive extreme ratings on Yelp. Wang (2010) also finds that more prolific Yelp reviewers have more friends on Yelp, receive more anonymous review votes per review, and display more compliment letters per review. These findings motivate us to explicitly model reviewers’ reputational concern on Yelp and allow elite and non-elite reviewers to place different value on Yelp popularity.10 6.3 Transparency and Aggregation Decisions Part of the motivation for this paper is that in almost every consumer review website, reviews are aggregated, prompting questions about how one should aggregate reviews. In practice, the most common way to aggregate reviews is using an arithmetic average, which is done by Amazon, Yelp, TripAdvisor, and many others. As we have highlighted in this paper, arithmetic average does not account for reviewer biases, reviewer heterogeneity, or changing quality. Another important caveat of our method is that there are reasons outside of our model that may prompt a review website to use an arithmetic average. For example, arithmetic averages are transparent and uncontroversial. If, for example, Yelp were to use optimal information aggregation, they may be accused of trying to help certain restaurants due to a conflict of interest (since Yelp also sells advertisements to restaurants). Hence, a consumer review website’s strategy might balance the informational benefits of optimal information aggregation against other incentives that may move them away from this standard, such as conflict of interest (or even the desire to avoid perceived conflict of interest). 6.4 Customized Recommendations Our paper has attempted to aggregate information into a single comparable signal of quality. Once this is done, it could be extended to then customize recommendations based on the readers horizontal prefences. For example, Netflix tailors recommendations based on other reviewers with similar tastes. This type of recommendation relies both on an understanding of underlying quality (as in this paper), as well as a sense of horizontal preferences of readers. 6.5 Text Analysis In this paper, we have focused on using only numerical data. However, a productive literature has begun to use text analysis to extract informational content from reviews. For examples, see Ghose and Ipeirotis (2011), Archak, Ghose and Ipeirotis (2011) and Ghose, Ipeirotis, and Li (2012). Ghose, Ipeirotis, and Li (2012) estimate the consumer demand model and argues that the ranking systems should be designed to reflect consumer demand besides price and star ratings. Ghose and Ipeirotis (2011) and Archak, Ghose, and Ipeirotis (2011) examine the 10 There is a large literature on social image and social influence, with most evidence demonstrated in lab or field experiments. For example, Ariely et al. (2009) show that social image is important for charity giving and private monetary incentives partially crowd out the image motivation. 26 impact of different product attributes and reviewer opinions on product sales, and propose a model to identify segments of text review that describe products’ multifaceted attributes. Although this is beyond the scope of the current paper, one could incorporate text analysis methods into optimal information aggregation. 6.6 Generalizing Our Approach In principle, the method offered in our paper could be applied to a variety of review systems. Implementing this could also be done in conjunction with the other considerations discussed above. Moreover, when generalizing our method, the relative importance of various factors in our model could vary by context. For example, quality change is not an issue for a fixed product such as book, movie, etc, reviewer heterogeneity could be much more important when expert and non-expert reviews are common but treated differently by the market (say for books and movies). The flexibility of our model allows it to be robust to this type of variation, while also allowing for new insights by applying the model to different settings. References Archak, Nikolay, Anindya Ghose, and Panagiotis G. Ipeirotis. "Deriving the pricing power of product features by mining consumer reviews." Management Science 57, no. 8 (2011): 1485-1509. Akaike, Hirotugu (1974). "A new look at the statistical model identification". IEEE Transactions on Automatic Control 19 (6): 716–723. Akerlof, George A. (1980) “A Theory of Social Custom, of Which Unemployment May Be One Consequence.” Quarterly Journal of Economics, 94(4): 749-75. Ariely, Dan, Anat Bracha, and Stephan Meirer. 2009. “Doing Good or Doing Well? Image Motivation and Monetary Incentives in Behaving Prosocially.” American Economic Review 99(1): 544-555. Banerjee, Abhijit V. "A simple model of herd behavior." The Quarterly Journal of Economics 107.3 (1992): 797-817. Bénabou, Roland, and Jean Tirole. 2006. “Incentives and Prosocial Behavior.” American Economic Review 96(5): 1652-78. Brown, Jennifer, Tanjim Hossain and John Morgan (2010) “Shrouded Attributes and Information Suppression: Evidence from the Field.” Quarterly Journal of Economics. 125(2): 859-876. Chan, Tat, Hai Che, Chunhua Wu, and Xianghua Lu, working paper “Social Network Learning: How User Generated Content on Review Website Influence Consumer Decisions” Chevalier, Judith A. and Dina Mayzlin. The effect of word of mouth on sales: Online book reviews. Journal of Marketing Research, 43(3):345–354, August 2006. Chen, Yan, F. Maxwell Harper, Joseph Konstan, and Sherry Xin Li (2010) “Social Comparison and Contributions to Online Communities: A Field Experiment on MovieLens.” American 27 Economic Review , 100(4): 1358-98. Dellarocas, Chrysanthos (2006). Strategic manipulation of internet opinion forums: Implications for consumers and firms. Management Science, 52(10):1577–1593. Duan, Wenjing, Bin Gu, and Andrew B. Whinston. "Do online reviews matter?—An empirical investigation of panel data." Decision Support Systems 45, no. 4 (2008): 1007-1016. Ghose, Anindya and Panagiotis G. Ipeirotis. Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics. IEEE Transactions on Knowledge and Data Engineering, 23(10):1498–1512, October 2011. Ghose, A., P. Ipeirotis, B. Li (forthcoming) “Designing Ranking Systems for Hotels on Travel Search Engines by Mining User-Generated and Crowd-Sourced Content” Marketing Science. Glazer, Jacob, Thomas G. McGuire, Zhun Cao, and Alan Zaslavsky. 2008. “Using Global Ratings of Health Plans to Improve the Quality of Health Care.” Journal of Health Economics, 27(5): 1182–95. Godes, David and Dina Mayzlin. Firm-created word-of-mout communication: Evidence from a field test. Marketing Science, 28(4):721–739, 2009. Goldenberg, Jacob, Barak Libai, and Eitan Muller. "The chilling effects of network externalities." International Journal of Research in Marketing 27, no. 1 (2010): 4-15. Alevy, Jonathan E., Michael S. Haigh, and John A. List. "Information cascades: Evidence from a field experiment with financial market professionals." The Journal of Finance 62, no. 1 (2007): 151-180. Hitt, Lorin and Xinxin Li. Self-selection and information role of online product reviews. Information Systems Research, 19:456–474, 2008. Hu, N.; Liu, L., and Zhang, J. (2008) “Do Online Reviews Affect Product Sales? The Role of Reviewer Characteristics and Temporal Effects” Information Technology Managment 9(3): 201-214. Hu, Mingqing and Bing Liu (2004). “Mining and summarizing customer reviews” Proceedings of the tenth ACM SIGKDD. Hu, Nan, Jie Zhang and Paul Pavlou (2009) “Overcoming the J-shaped distribution of product reviews”, Communication ACM. Li, Xinxin and Lorin Hitt (2008) “Self-Selection and Information Role of Online Product Reviews,” Information Systems Research 19(4): 456-474. Luca, Michael and Jonathan Smith (forthcoming) “Salience in Quality Disclosure: Evidence from The US News College Rankings,” Journal of Economics & Management Strategy. Luca, Michael (2011) "Reviews, Reputation, and Revenue: The Case of Yelp.com." Harvard Business School working paper . Mayzlin, Dina. "Promotional chat on the Internet." Marketing Science 25, no. 2 (2006): 155-163. Miller, Nolan, Paul Resnick, and Richard J. Zeckhauser (2005). “Eliciting Informative 28 Feedback: The Peer- Prediction Method.” Management Science, 51(9): 1359–73. Netzer, Oded, Ronen Feldman, Jacob Goldenberg, and Moshe Fresko (2012). “Mine Your Own Business: Market Structure Surveillance Through Text Mining.” Marketing Science 31 (3). Pope, Devin (2009). "Reacting to Rankings: Evidence from ’America’s Best Hospitals’," Journal of Health Economics, Vol. 28, No. 6, 1154-1165. Wang, Qingliang, Khim Yong Goh, Xianghua Lu, 2012. “How does user generated content influence consumers’ new product exploration and choice diversity? An empirical analysis of product reviews and consumer variety seeking behaviors,” working paper. Wang, Zhongmin “Anonymity, Social Image, and the Competition for Volunteers: A Case Study of the Online Market for Reviews” The B.E. Journal of Economic Analysis & Policy, 2010. Appendix A: Derive E(µrt |srt1 , ...srtn ) For restaurant r, denote the prior belief of µrtn right before the realization of the nth signal as n|n 1 (µrtn ) = f (µrtn |srt1 , ...srtn 1 ) and we assume that the first reviewer uses an uninformative prior 2 µ1|0 = 0, ↵1|0 = W, W arbitrarily large Denote the posterior belief of µrtn after observing srtn as hn|n (µrtn ) = f (µrtn |srt1 , ...srtn ) Hence hn|n (µrtn ) = f (µrtn |srt1 , ...srtn ) = f (µrtn , srt1 , ...srtn ) f (srt1 , ...srtn ) ⌥f (µrtn , srt1 , ...srtn ) =f (srtn |µrtn , srt1 , ...srtn 1 )f (µrtn , srt1 , ...srtn 1 ) =f (srtn |µrtn , srt1 , ...srtn 1 )f (µrtn |srt1 , ...srtn 1 )f (srt1 , ...srtn 1 ) ⌥f (srtn |µrtn )f (µrtn |srt1 , ...srtn 1 ) =f (srtn |µrtn ) n|n 1 (µrtn ) where f (srtn |µrtn , srt1 , ...srtn 1 ) = f (srtn |µrtn ) comes from the assumption that srtn is independent of past signals conditional on µrtn . In the above formula, the prior belief of µrtn given the realization of {srt1 , ..., srtn 1 }, or 29 n|n 1 (µrtn ), depends on the posterior belief of µrtn 1 , hn process from µrtn 1 n|n 1 (µrtn ) to µrtn , denoted as g(µn |µn = g(µn |µn Given the normality of 1 )f (µrtn n|n 1 , mal. In addition, denote µn|n and 1 1 ). normal probability density function pn|n 1 ) and the evolution Hence, |srt1 , ...srtn 1 ) = g(µn |µn f (srtn |µrtn ) and g(µn |µn 2 ↵n|n 1|n 1 (µrtn 1 ), 1 )hn 1|n 1 (µrtn 1 ) hn|n (µrtn ) is distributed nor- as the mean and variance for random variable with 1 (µrtn ), µn|n 1 2 and ↵n|n 1 are the mean and variance of random variable with normal pdf hn|n (µrtn ). After combining terms in the derivation of pn|n 1 (µrtn ) and hn|n (µrtn ), the mean and variance evolves according to the following rule: µn|n = µn|n = 2 ↵n|n = 1+ 2 ↵n|n 2 ↵n|n 2 ↵n|n 1 1 2 ↵n|n 1 + ↵n2 s + 2 n 1 + ↵n 2 ↵n2 ↵n|n 1 2 2 ↵n|n + ↵ n 1 (sn µn|n 1) ↵n2 µ 2 2 n|n ↵n|n 1 + ↵n 1 µn+1|n = µn|n 2 2 ↵n+1|n = ↵n|n + (tn+1 tn )↵⇥2 Hence, we can deduct the beliefs from the initial prior, µ1|0 = 0 2 ↵1|0 = W > 0 and arbitrarily large µ1|1 = s1 2 ↵1|1 = ↵12 µ2|1 = s1 2 ↵2|1 = ↵12 + (t2 µ2|2 = 2 ↵2|2 = t1 )↵⇥2 ↵12 + (t2 ↵12 + ↵22 + (t2 ↵22 (↵12 + (t2 ↵12 + ↵22 + (t2 t1 )↵⇥2 2 s2 + t1 )↵⇥ ↵22 ↵12 + ↵22 + (t2 t1 )↵⇥2 s1 t1 )↵⇥2 ) t1 )↵⇥2 µ3|2 = µ2|2 2 ↵3|2 = ↵22 (↵12 + (t2 ↵12 + ↵22 + (t2 t1 )↵⇥2 ) t1 )↵⇥2 + (t3 t2 )↵⇥2 ... E(µrtn |srt1 , ...srtn ) = µn|n is derived recursively following the above formulation. 30 Appendix B: Solve {srt1 , ...srtn } from {xrt1 , ...xrtn } according to the following recursive formula: x 1 = s 1 + ⌥1 s 1 = x1 ⌥1 2 ↵2|1 ↵22 x2 = ⌦2 2 µ2|1 + ⌦2 2 s2 + (1 ↵2|1 + ↵22 ↵2|1 + ↵22 ↵2 = ⌦2 2 2 2 µ2|1 + [1 ↵2|1 + ↵2 1 s2 = [1 (1 2 ⇤2|1 2 ⇤2|1 +⇤22 2 ↵2|1 (1 [x2 2 + ↵2 ↵2|1 2 ⌥2 ⌦2 )⌦2 ] ⌦2 )s2 + ⌥2 )⌦2 ]s2 + ⌥2 ↵22 2 + ↵ 2 µ2|1 ] ↵2|1 2 ... 1 sn = [1 (1 2 ⇤n|n 2 ⇤n|n 1 [xn 1 )⌦n ] +⇤ 2 n 31 ⌥n ⌦n ↵n2 µ 2 2 n|n ↵n|n 1 + ↵n 1 ].