Optimal Aggregation of Consumer Ratings Tables

Table 1: Summary Statistics Variable Mean Med. Min. Restaurant Characteristics Reviews per Restaurant 32.85 14.00 1.00 Reviews per Day 0.16 0.03 0.00 154.75 79.00 0.00 Days between 1st and 2nd Review 33.96 20.00 0.00 Days between 11st and 12nd Review st nd 20.63 13.00 0.00 Days between 21 and 22 Review Reviewer Characteristics Rating by Elite by Non-elite Reviews per reviewer by Elite by Non-elite Reviews per Day by Elite by Non-elite Reviewer-Restaurant Matching Distanceb by Elite by Non-elite Reviewer Taste for Varietyc by Elite by Non-elite 3.74 3.72 3.75 7.18 24.49 5.35 0.12 0.15 0.12 12.18 11.26 12.62 1.10 1.11 1.09 4.00 4.00 4.00 2.00 6.00 2.00 0.17 0.22 0.16 8.51 7.47 9.00 1.11 1.12 1.10 1.00 1.00 1.00 1.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Max. Std. Dev. N.a 698.00 5.00 1,544.00 519.00 234.00 50.20 0.33 199.95 41.71 25.27 4,101 4,101 3,651 2,199 1,649 5.00 5.00 5.00 453.00 350.00 453.00 1.52 1.30 1.52 108.00 108.00 103.73 2.60 2.60 2.52 1.14 1.10 1.18 17.25 39.23 11.49 0.07 0.10 0.07 11.45 10.77 11.74 0.24 0.17 0.27 134,730 43,781 90,949 18,778 1,788 16,990 18,778 1,788 16,990 134,730 43,781 90,949 103,835 40,521 63,314 a There are 4,101 restaurants with a total of 134,730 reviews written by 18,778 reviewers in our sample. b Reviewer-restaurant matching distance quantifies the quality of the match between a reviewer and a restaurant, calculated using information on restaurants the reviewer has previously rated. See page 7 for the formula and more thorough discussion. c Reviewer taste for variety quantifies the reviewer’s preference for variety in restaurants, calculated using information on restaurants the reviewer has previously rated. See page 7 for the formula and more thorough discussion. Table 2: What Explains the Variance of Yelp Ratings? Model Variance Explained (R2 ) Reviewer FE 0.2329 Restaurant FE 0.2086 Reviewer FE & Restaurant FE 0.3595 Reviewer FE & Restaurant FE & Year FE 0.3595 Reviewer FE & Restaurant FE & Year FE 0.3749 & Matching & Variety Notes: 1 This table presents the R2 from linear regressions of Yelp ratings on the fixed effects and variables indicated in each row. 2 There are only a few observations in 2004 and 2010. Whenever the year fixed effect is added, 2005 year fixed effect was applied to the 2004 observations and 2009 year fixed effect was applied for the 2010 observations. Table 3: Does the Variability of Ratings Decline over Time? 2 Model:a ⇥[ Dri,elite + ⇤ri,yr ri,yr = 0 + 1 Dri,elite + 2 Nri + 3 Nri elite b Dri Nri c (100s) elite Dri Nri (100s) constant N -12.000⇤⇤⇤ -0.021⇤⇤ -0.009 88.000⇤⇤⇤ 134,730 (0.940) (0.007) (0.012) (0.581) a ⇤[ ri,yr is the residual from regression Ratingri,year = µr + b elite Dri c Nri indicates that reviewer i writes the N th review on restaurant r. i + ⇥year + ⇤ri,year is the dummy variable for elite reviewer. Standard errors in parentheses. ⇤ p < 0.05, ⇤⇤ p < 0.01, ⇤⇤⇤ p < 0.001 Table 4: Restaurant Ratings are Serially Correlated ⇥r,i \ 1,yr ⇥r,i \ 2,yr ⇥r,i \ 3,yr ⇥r,i \ 4,yr ⇥\ r,i 5yr Pk Model:a ⇥[ ri,yr (1) 0.0428⇤⇤⇤ (0.0029) 0.0299⇤⇤⇤ (0.0029) 0.0213⇤⇤⇤ (0.0029) 0.0151⇤⇤⇤ (0.0029) 0.0126⇤⇤⇤ (0.0029) = -0.00629⇤ (0.00266) 117,536 -0.00782⇤⇤ (0.00269) 114,742 ⇥r,i \ 5,yr s=1 \ s ⇥r,i s,yr (2) 0.0433⇤⇤⇤ (0.0030) 0.0300⇤⇤⇤ (0.0030) 0.0208⇤⇤⇤ (0.0030) 0.0146⇤⇤⇤ (0.0030) 0.0117⇤⇤⇤ (0.0030) 0.0087⇤⇤ (0.0030) ⇥r,i \ 6,yr + ⌅ri,yr (3) 0.0429⇤⇤⇤ (0.0030) 0.0299⇤⇤⇤ (0.0030) 0.0209⇤⇤⇤ (0.0030) 0.0145⇤⇤⇤ (0.0030) 0.0111⇤⇤⇤ (0.0030) 0.0081⇤⇤ (0.0030) 0.00991⇤⇤⇤ (0.00300) ⇥r,i \ 7,yr Constant Observations -0.00856⇤⇤ (0.00272) 112,067 (4) 0.0423⇤⇤⇤ (0.0030) 0.0311⇤⇤⇤ (0.0030) 0.0213⇤⇤⇤ (0.0030) 0.0148⇤⇤⇤ (0.0030) 0.0110⇤⇤⇤ (0.0030) 0.0084⇤⇤ (0.0030) 0.00996⇤⇤ (0.00303) 0.00312 (0.00303) -0.00971⇤⇤⇤ (0.00275) 109,505 Notes: This table estimates the degree of serial correlation of ratings within a restaurant. a ⇤[ ri,yr is the residual from regressing Ratingri,year = µr + i + ⇥year + ⇤ri,year . To obtain sequential correlation of the residuals, we regress the residuals on their lags ⇤ri \ s,yr , where s is the number of lag. ⇤⇤⇤ Standard errors in parentheses. ⇤ p < 0.05, ⇤⇤ p < 0.01, ⇤⇤⇤ p < 0.001. Table 5: Does Matching Improve Over Time? For Restaurants For Reviewers (1) (2) (3) (4) Restaurant’s nth Review (Restaurant’s nth Review)2 (Restaurant’s nth Review)3 Matching Distancea 0.00170 (0.00117) -0.00002 (0.00001) 1.06e-08 (8.00e-09) Taste for Varietyb -0.00025 (0.00006) 1.17e-6 (3.25e-7) -1.54e-09 (4.01e-10) Reviewer’s nth Review (Reviewer’s nth Review)2 (Reviewer’s nth Review)3 Constant Observations 12.13 (0.03590) 134,730 1.104 (0.00157) 103,835 Matching Distancea Taste for Varietyb -0.06700 (0.00140) 0.00045 (0.00001) 7.57e-7 (2.14e-08) 12.57 (0.02330) 103,835 0.00169 (0.00006) -0.00001 (4.46e-7) 1.95e-08 (8.35e-10) 1.066 (0.00103) 103,835 Notes: The sample sizes of regressions specified in column (2), (3) and (4) are smaller since we dropped the first reviews written by a reviewer. Because reviewer does not have any review history when she reviews for the first time, we set the two measures at their overall sample mean. a Reviewer-restaurant matching distance quantifies the quality of the match between a reviewer and a restaurant, calculated using information on restaurants the reviewer has previously rated. See page 7 for the formula and more thorough discussion. b Reviewer taste for variety quantifies the reviewer’s preference for variety in restaurants, calculated using information on restaurants the reviewer has previously rated. See page 7 for the formula and more thorough discussion. Standard errors in parentheses. p < 0.05, p < 0.01, p < 0.001 Table 6: MLE Estimates: Signal Precision and Popularity Concerns ⇤e (1) Same ⇤, ⇥ (2) Different ⇤, ⇥ (3) Quarterly Quality Change 1.2218⇤⇤⇤ (0.0210) 1.1753⇤⇤⇤ (0.0210) 1.2350⇤⇤⇤ 0.0147 0.1718⇤⇤⇤ (0.0007) 0.2430⇤⇤⇤ (0.0141) 0.1362⇤⇤⇤ (0.0110) 0.0100 (0.0059) 0.9293⇤⇤⇤ (0.0199) 0.9850⇤⇤⇤ (0.0156) 0.1452⇤⇤⇤ (0.0038) 0.0454⇤⇤⇤ (0.0215) 0.0821⇤⇤⇤ (0.0181) -0.0061 (0.0059) -193,339 122,473 -192,538 122,473 -192,085 122,473 ⇤ne ⇤ ⇥e ⇥ne ( ( ( ne )0 e ne )Age e ne )M atchD e ( e ne )T asteV ar ( e ne )F reqRev ( e ne )N umRev (µ + (µ + ne )Age ne )M atchD (µ + ne )T asteV ar (µ + ne )F reqRev (µ + ne )N umRev Log Likelihood N (4) Full Model (Quarterly Quality Change) 0.9101⇤⇤⇤ (0.0196) 0.9546⇤⇤⇤ (0.0151) 0.1345⇤⇤⇤ (0.0039) 0.0230⇤⇤⇤ (0.0220) 0.1182⇤⇤⇤ (0.0187) 0.0007 (0.0223) 3.8 ⇥ 10 5⇤⇤⇤ (1.3 ⇥ 10 5 ) 0.0002 (0.0005) -0.0025 (0.0064) 0.0647⇤⇤ (0.0319) 0.0006⇤⇤⇤ (0.00015) 0.0002⇤⇤⇤ (1.3 ⇥ 10 5 ) 0.0042⇤⇤⇤ (0.0005) 0.0224⇤⇤⇤ (0.0029) 0.0348 (0.0218) 0.0007⇤⇤⇤ (0.0001) -191,829 122,473 Notes: 1. “e” and “ne” in the subscript indicate elite and non-elite status respectively. 2. Parameters in row (7) to (16) represent the effects of review-restaurant matching and reviewer attributes on sum of restaurant quality and non-elite stringency ( ne + µ) and stringency difference between elite and non-elite reviewers. The subscripts that represent review-restaurant matching and reviewer attributes are reviewer-restaurant matching distance (M atchD), reviewer taste for variety (T asteV ar), number of reviews written by the reviewer per day (F reqRev), and total number of reviews written by the reviewer (N umRev). Standard errors in parentheses. p < 0.05, p < 0.01, p < 0.001. Table 7: MLE Estimates with Changing Restaurant Quality ⇤e ⇤ne ⇤ ⇥e ⇥ne ( ( ( ne )0 e ne )Age e ne )M atchD e ( e ne )T asteV ar ( e ne )F reqRev ( e ne )N umRev (µ + (µ + ne )Age ne )M atchD (µ + ne )T asteV ar (µ + ne )F reqRev (µ + ne )N umRev Log Likelihood N (1) Full Model (Quarterly Quality Change) 0.9101⇤⇤⇤ (0.0196) 0.9546⇤⇤⇤ (0.0151) 0.1345⇤⇤⇤ (0.0039) 0.0230⇤⇤⇤ (0.0220) 0.1182⇤⇤⇤ (0.0187) 0.0007 (0.0223) 3.8 ⇥ 10 5⇤⇤⇤ (1.3 ⇥ 10 5 ) 0.0002 (0.0005) -0.0025 (0.0064) 0.0647⇤⇤ (0.0319) 0.0006⇤⇤⇤ (0.00015) 0.0002⇤⇤⇤ (1.3 ⇥ 10 5 ) 0.0042⇤⇤⇤ (0.0005) 0.0224⇤⇤⇤ (0.0029) 0.0348 (0.0218) 0.0007⇤⇤⇤ (0.0001) -191,829 (2) Full Model (Half-yearly Quality Change) 0.9365⇤⇤⇤ (0.0196) 0.9778⇤⇤⇤ (0.0150) 0.1736⇤⇤⇤ (0.0051) 0.0499⇤⇤⇤ (0.0209) 0.0922⇤⇤⇤ (0.0176) -0..00057 (0.0223) 3.9 ⇥ 10 5⇤⇤⇤ (1.3 ⇥ 10 5 ) 0.0001 (0.0005) -0.0025 (0.0064) 0.0657⇤⇤ (0.0319) 0.0006 (0.00055) 0.0002⇤⇤⇤ (1.3 ⇥ 10 5 ) 0.0040⇤⇤⇤ (0.0005) 0.0233⇤⇤⇤ (0.0029) 0.0347 (0.0218) 0.0007 (-0.0007) -191,870 (3) Full Model (Monthly Quality Change) 0.8983⇤⇤⇤ (0.0196) 0.9440⇤⇤⇤ (0.0153) 0.0810⇤⇤⇤ (0.0023) 0.0103⇤⇤⇤ (0.0227) 0.1307⇤⇤⇤ (0.0193) 0.0007 (0.0223) 3.8 ⇥ 10 5⇤⇤⇤ (1.3 ⇥ 10 5 ) 0.0002 (0.0005) -0.0024 (0.0064) 0.0640⇤⇤ (0.032) 0.0006⇤⇤⇤ (0.00015) -0.0003⇤⇤⇤ (1.4 ⇥ 10 5 ) 0.0040⇤⇤⇤ (0.0005) 0.0233⇤⇤⇤ (0.0029) 0.0353 (0.0218) 0.0007⇤⇤⇤ (-0.0001) -191,814 122,473 122,473 122,473 Notes: 1. Full model specification allows random walk of restaurant quality in different frequency, and both restaurant quality and reviewer stringency to vary flexibly with reviewer-restaurant matching matrices and reviewer attributes. 2. “e” and “ne” in the subscript indicate elite and non-elite status respectively. 3. Parameters in row (7) to (16) represent the effects of review-restaurant matching and reviewer attributes on sum of restaurant quality and non-elite stringency ( ne + µ) and stringency difference between elite and non-elite reviewers. The subscripts that represent review-restaurant matching and reviewer attributes are reviewer-restaurant matching distance (M atchD), reviewer taste for variety (T asteV ar), number of reviews written by the reviewer per day (F reqRev), and total number of reviews written by the reviewer (N umRev). Standard errors in parentheses. p < 0.05, p < 0.01, p < 0.001. Table 8: Optimal and Simple Average Algorithm Applied on the Real Data µ Update Frequency % µ < 0.15 % µ < 0.25 % µ < 0.35 % µ > 0.15 % µ > 0.25 % µ > 0.35 0.33% 0.18% 0.39% 0.06% 0.06% 0.09% 0.03% 0% 0.03% 24.39% 23.02% 24.57% 9.00% 7.62% 9.72% 2.87% 2.30% 3.32% (i) Restaurant quality unaffected by restaurant age Quarterly Half-yearly Monthly 54.08% 53.48% 54.17% 28.40% 26.43% 29.21% 13.42% 11.09% 13.69% (ii) Restaurant quality affected only by restaurant age Quarterly Half-yearly Monthly Notes: 1. 2.57% 1.82% 2.99% µ ⇥ µsimple 0.54% 0.39% 0.63% 0.09% 0% 0.15% µoptimal calculates the difference between simple average µsimple and optimal average µoptimal when a restaurant receives its last rating as of the end of our sample. %( µ > x) calculates the percentage of restaurants having optimal and simple average difference greater than x. 2. Given that in ⇥(µrn + quality µrn and reviewer age only affects ne rn ne rn )/⇥n, bias ne rn , we cannot separately identify the impact of restaurant age n on restaurant we calculate µsimple µoptimal in the model assuming that restaurant (in panel (i)), and assuming that restaurant age only affects µrn (in panel (ii)) 0 0 .1 .1 Kernel Density .2 .3 Kernel Density .2 .3 .4 .4 .5 .5 Figure 1: The Distribution of Ratings by Elite Status −4 −2 0 2 Rating minus Restaurant Past Mean Elite Reviewers Ratingrn −4 −2 0 2 Rating Minus Restaurant Future Mean Non−elite Reviewers Notes. 1. The figure on the left plots Ratingrn BF Rating rn 4 is the arithmetic mean of all past n Elite Reviewers 4 Non−Elite Reviwers BF Rating rn , where Ratingrn is the nth rating on restaurant r, and 1 ratings on restaurant r before n. Similarly, figure on the right plots AF AF Rating rn , where Ratingrn is the nth rating on restaurant r, and Rating rn is the arithmetic mean of all future ratings on restaurant r until the end of the sample. 2. This figure shows the distribution of ratings relative to the restaurant mean. Ratings by elite reviewers tend to be closer to a restaurant’s average rating, and have shorter tails. Figure 2: Restaurants Experience a “Chilling Effect” -.05 Fitted Residuals 0 .05 .1 Fractional Polynomial Smooth of Residuals 0 20 40 Nth Ratings on the Restaurant 95% CI 60 Fitted Residuals Notes. This figure shows the trend of ratings within a restaurant over time. Restaurants begin with more favorable reviews, which deteriorate over time. It plots the fractional polynomial of the restaurant residual on the sequence of reviews. Residuals are obtained from regression Ratingrn,year = µr + year + ⇥rn,year . Figure 3: Comparing Optimal and Simple Averages: Reviewers with Popularity Concern Parameters ⇥ Restaurant Quality (Left) = = 1 ⇥ = ⇥ = 1 Quality fixed at µ = 3 e ne e ne (Right) ⇥e = ⇥ne = 1 Quality fixed at µ = 3 e = ne = 0.6 Optimal Vs. Simple Average, ρ = 1 Simple Average Vs. Optimal Average,ρ = 0.6 5 5 95% C.I. of Optimal Average 95% C.I. of Simple Average 4 4 3.5 3.5 3 3 2.5 2.5 2 2 1.5 1.5 1 0 10 20 30 40 50 95% C.I. of Optimal Average 95% C.I. of Simple Average 4.5 Rating Rating 4.5 1 60 0 10 nth rating on the restaurant 20 30 40 50 60 nth rating on the restaurant Notes: The above figures plot the simulated 95% confidence interval for the average ratings that would occur for a restaurant at a given quality level. When reviewers have popularity concern, arithmetic and optimal averages are unbiased estimates for true quality. But relative to arithmetic average, optimal aggregation allows ratings to converge to the true quality more quickly. And the difference in converging speeds increases with elite reviewers’ popularity concern relative to that of non-elite reviewers. Figure 4: Comparing Optimal and Simple Averages: Reviewers with Different Precisions Parameter ⇥ Restaurant Quality (Left) ⇥e = 0.6,⇥ne = 1 Quality fixed at µ = 3 e = ne = 0 ⇥e = 0.3,⇥ne = 1 Quality fixed at µ = 3 (Right) e = ne = 0 Simple Average Vs. Optimal Average,σne = 1,σe=0.3 Simple Average Vs. Optimal Average,σne = 1,σe=0.6 5 4.5 95% C.I. of Optimal Average 95% C.I. of Simple Average 4.5 95% C.I. of Optimal Average 95% C.I. of Simple Average 4 4 3.5 Rating Rating 3.5 3 3 2.5 2.5 2 2 1.5 1 0 10 20 30 40 nth rating on restaurant 50 60 1.5 0 10 20 30 40 50 60 nth rating on restaurant Notes: The above figures plot the simulated 95% confidence interval for the average ratings that would occur for a restaurant at a given quality level. When reviewers differ in precision, both arithmetic and optimal averages are unbiased estimates for true quality. But relative to arithmetic average, optimal aggregation allows ratings to converge to the true quality more quickly. The difference in converging speed increases with elite reviewers’ precision relative to that of non-elite reviewers. Figure 5: Comparing Optimal and Simple Averages: Martingale Quality Change Quality Update Frequency Quarterly Quarterly Monthly ⇥ (Top Left) (Top Right) (Bottom) e e e = = = =0 =0 =0 ne ne ne ⇥e = ⇥ne = 0.6 ⇥e = ⇥ne = 0.6 ⇥e = ⇥ne = 0.6 Optimal Vs. Simple Average, (Quality change quarterly, σe = σne = 0.6,σµ = 0.5) 0.5 Mean Absolute Error of Optimal Average Mean Sqaured Error of Optimal Average Mean Absolute Error of Simple Average Mean Sqaured Error of Simple Average 0.45 0.4 0.4 0.35 0.35 0.3 0.3 0.25 0.25 0.2 0.2 0.15 0.15 0.1 0.1 0.05 0.05 0 10 20 th 30 40 50 Mean Absolute Error of Optimal Average Mean Sqaured Error of Optimal Average Mean Absolute Error of Simple Average Mean Sqaured Error of Simple Average 0.45 Rating Rating Quality ⇥ = 0.5 ⇥ = 0.2 ⇥ = 0.2 Optimal Vs. Simple Average, (Quality change quarterly, σe = σne = 0.6,σµ = 0.2) 0.5 0 Variance of 0 60 0 10 20 th n rating on the restaurant 30 40 50 60 n rating on the restaurant Optimal Vs. Simple Average, (Monthly Quality Change, σe = σne = 0.6,σµ = 0.2) 0.5 Mean Absolute Error of Optimal Average Mean Sqaured Error of Optimal Average Mean Absolute Error of Simple Average Mean Sqaured Error of Simple Average 0.45 0.4 0.35 Rating 0.3 0.25 0.2 0.15 0.1 0.05 0 0 10 20 th 30 40 50 60 n rating on the restaurant Notes: The above figures plot the mean absolute error and the mean squared error of optimal and simple average on restaurants with quality updates following a martingale process. The error on each simulated path is calculated as the difference between the averages and the true quality of the restaurant when the nth review is written. The figures show that relative to arithmetic average, errors of optimal aggregation shrink faster. And the difference in this speed increases when qualities of restaurants change more often, and when the variance of incremental quality change is larger. Figure 6: How Quickly Do Ratings Adjust to Changes in Restaurant Quality? e = ne ⇥ ⇥e = ⇥ne = 0.6 =0 Quality Update Frequency Quarterly Variance of Quality ⇥ = 0.5 Optimal Vs. Simple Average, (Quarterly Quality Change, σ = σ = 0.6,σ = 0.5) e ne ξ 4.5 95% C.I. of Optimal Average 95% C.I. of Simple Average Mean of Optimal Average Mean of Simple Average True Path of Restaurant Quality 4 Rating 3.5 3 2.5 2 1.5 0 10 20 30 40 50 60 th n rating on the restaurant Notes: The above figures plot the simulated mean and 95% confidence interval for the average ratings that would occur for a single restaurant with its quality realized from the quarterly quality change martingale model. This quality path features restaurant’s quality dropping before it receives its 20th rating, and jumping before it receives its 40th rating. Relative to arithmetic average, the optimal aggregation adapts to the change in restaurant’s true quality more quickly. Since the optimal aggregation algorithm only gives weights to recent ratings, when fixing the review frequency, its standard error shrinks slower than arithmetic average that gives equal weights to all historical ratings. Figure 7: Simulated Ratings when Reviewers are Biased Parameters e , ne Value 0 ⇥e , ⇥ne 1 Quality0 3 Parameters Value ⇥(µ+ e ) -0.0002 ⇥Restaurat Age ⇥(µ+ e ) 0.0042 ⇥M atch Distance ⇥(µ+ e ) ⇥Reviewer T aste T o V ariety-0.0224 ⇥(µ+ e ) -0.0348 ⇥Reviewer F requency Optimal Vs. Simple Average Fixed Restaurant Quality and Changing Reviewer Biases 0.00004 0.0647 0.0006 5 95% C.I. of Optimal Average 95% C.I. of Simple Average Mean of Optimal Average Mean of Simple Average 4.5 4 4 3.5 3.5 3 3 2.5 2.5 2 2 1.5 1.5 0 10 20 30 40 nth rating on the restaurant 50 60 95% C.I. of Optimal Average 95% C.I. of Simple Average Mean of Optimal Average Mean of Simple Average 4.5 Rating Rating Value -0.0007 ⇥(µ+ e ) ⇥Reviewer Review # ⇥( e ne ) ⇥Restaurat Age ⇥( e ne ) ⇥Reiviewer F requency ⇥( e ne ) ⇥Reviewer Review # Optimal Vs. Simple Average Worsening Restaurant Quality and Changing Reviewer Biases 5 1 Parameters 1 0 10 20 30 40 50 60 nth rating on the restaurant Notes: The above figures plot the simulated mean and 95% confidence interval for the average ratings that would occur for restaurants with biased reviewers. The figure on the left assumes that restaurants have fixed quality at 3, and reviewers’ bias is trending downwards with restaurant age. The figure on the right assumes that the restaurants have quality trending downwards with restaurant age, and the reviewer bias is unaffected by restaurant age. In both cases, we assume that reviewers perfectly acknowledge other reviewers’ biases and the common restaurant quality trend. So in both cases, optimal aggregation is an unbiased estimate for true quality while the arithmetic average is biased without correcting the review bias. Figure 8: Comparing Optimal and Simple Averages: Full Model Specification Optimal Vs. Simple Average, (Full Model Specifications, Downward Quality Trend) 0.9 Mean Absolute Error of Optimal Average Mean Sqaured Error of Optimal Average Mean Absolute Error of Simple Average Mean Sqaured Error of Simple Average 0.8 0.7 Rating 0.6 0.5 0.4 0.3 0.2 0.1 0 0 10 20 30 40 50 60 70 80 nth rating on the restaurant Notes: The above figure plots the mean absolute error and the mean squared error of optimal and simple average on restaurants with ratings generated from the full model with estimated parameters shown in Table 7. The error on each simulated path is calculated as the difference between the averages and the true quality of the restaurant when the nth review is written. The figure shows that relative to arithmetic average, errors of optimal aggregation shrink faster. And this difference is the same if we assume the downward rating trend is due to reducing quality or biased consumers. Figure 9: Optimal and Simple Average Algorithm Applied on the Real Data (Top) (Middle) (Bottom) Quarterly Quality Change Half-yearly Quality Change Monthly Quality Change Difference Between Simple and Optimal Average Full Model, Quality Change Half Yearly Assess Difference Between Simple and Optimal Average (Full Model, Half Yearly Quality Change) 1200 Mean of difference between simple and optimal average 95% C.I. of difference between simple and optimal average 0.2 1000 0.1 0 # of restaurants 800 Rating −0.1 −0.2 −0.3 600 400 −0.4 200 −0.5 0 10 20 30 40 50 60 70 80 90 0 100 0 0.1 nth rating on the restaurant 0.2 0.3 0.4 0.5 0.6 % of within restaurant ratings having |µ 0.7 −µ opt Difference Between Simple and Optimal Average Full Model, Quarterly Quality Change 0.8 0.9 |>0.15 1 simple Assess Difference Between Simple and Optimal Average (Full Model, Quarterly Quality Change) 1200 Mean of difference between simple and optimal average 95% C.I. of difference between simple and optimal average 0.2 1000 0.1 0 # of restaurants 800 Rating −0.1 −0.2 −0.3 600 400 −0.4 200 −0.5 −0.6 0 10 20 30 40 50 60 70 80 90 0 100 0 0.1 nth rating on the restaurant Difference Between Simple and Optimal Average Full Model, Monthly Quality Change 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 % of within restaurant ratings having |µopt−µsimple|>0.15 1 Assess Difference Between Simple and Optimal Average (Full Model, Monthly Quality Change) 1200 Mean of difference between simple and optimal average 95% C.I. of difference between simple and optimal average 0.2 1000 0.1 0 # of restaurants 800 Rating −0.1 −0.2 −0.3 600 400 −0.4 200 −0.5 −0.6 0 10 20 30 40 50 60 nth rating on the restaurant 70 80 90 100 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 % of within restaurant ratings having |µopt−µsimple|>0.15 1 Notes: 1. Figures on the left column plot the trend of mean and 95% confidence interval for µsimple rn the right column plot the frequency of restaurants that have proportions of ratings satisfying µoptimal . Figures on rn optimal |µrn µsimple | > 0.15. rn 2. The model we present here assumes that there exists a reviewer “chilling” effect instead of downward trend of restaurant quality. Figure 10: Comparing Optimal and Simple Averages: Temporary Shocks to the Quality (Left) (Right) (Bottom) Quality Shock µ1 = 1.5 µ3 = 1.5 µ5 = 1.5 Quality Update Frequency Quarterly Quarterly Quarterly ⇥ e e e = = = ne ne ne =0 =0 =0 ⇥e = ⇥ne = 0.6 ⇥e = ⇥ne = 0.6 ⇥e = ⇥ne = 0.6 3.4 3.2 3.2 3 3 2.8 2.8 2.6 2.6 2.4 2.4 2.2 2.2 2 2 1.8 1.8 Mean of Optimal Average Mean of Simple Average True Path of Restaurant Quality 1.6 1.4 ⇥ = 0.5 ⇥ = 0.5 ⇥ = 0.5 Simple Vs. Optimal Average rd (Quality Shock at the 3 Rating) 3.4 Rating Rating Simple Vs. Optimal Average st (Quality Shock at the 1 Rating) Update Noise 0 10 20 30 40 50 Mean of Optimal Average Mean of Simple Average True Path of Restaurant Quality 1.6 1.4 60 0 10 20 nth rating on restaurant 30 40 50 60 nth rating on restaurant Simple Vs. Optimal Average th (Quality Shock at the 5 Rating) 3.4 3.2 3 2.8 Rating 2.6 2.4 2.2 2 1.8 Mean of Optimal Average Mean of Simple Average True Path of Restaurant Quality 1.6 1.4 0 10 20 30 40 50 60 nth rating on restaurant Notes: The above figures plot the simulated mean of the average ratings for a single restaurant that follows the martingale model of quarterly quality change, but experiences a temporary quality shock. Focusing on the temporary quality shock, the optimal aggregation is more responsive to temporary quality shock and converges back to the true quality in a faster rate. This is due to the fact that optimal aggregation gives higher weight to more recent reviews. In comparison, simple average smooths the shock by giving equal weight to all historical ratings. Figure 11: Comparing Optimal and Simple Averages: Temporary Shocks to the Signals (Left) (Right) (Bottom) Signal Shock µ1 = 1.5 µ3 = 1.5 µ5 = 1.5 Quality Update Frequency Quarterly Quarterly Quarterly ⇥ e e e = = = ne ne ne =0 =0 =0 ⇥e = ⇥ne = 0.6 ⇥e = ⇥ne = 0.6 ⇥e = ⇥ne = 0.6 Optimal Vs. Simple Average, st (Signal Shock at the 1 Rating) 3.5 Mean of Optimal Average Mean of Simple Average True Path of Restaurant Quality Mean of Optimal Average Mean of Simple Average True Path of Restaurant Quality 3 Rating 3 Rating ⇥ = 0.5 ⇥ = 0.5 ⇥ = 0.5 Optimal Vs. Simple Average, rd (Signal Shock at the 3 Rating) 3.5 2.5 2 1.5 Update Noise 2.5 2 0 10 20 30 40 50 1.5 60 0 10 20 nth rating on restaurant 30 40 50 60 nth rating on restaurant Optimal Vs. Simple Average, th (Signal Shock at the 5 Rating) 3.5 Mean of Optimal Average Mean of Simple Average True Path of Restaurant Quality Rating 3 2.5 2 1.5 0 10 20 30 40 50 60 nth rating on restaurant Notes: The above figures plot the simulated mean of the average ratings for a single restaurant that follows the martingale model of quarterly quality change, but there is a temporary shock to the signal (We can think of this happening in the situation when a reviewer misreports her signal). Both aggregating algorithms weight past ratings, and are affected by the rating based on the shoecked signal. But compared with arithmetic mean, optimal aggregation “forgets” about earlier ratings and converges back to the true quality in a faster rate.