Multitasking, Learning, and Incentives - A Cautionary Tale

Multitasking, Learning, and Incentives: A Cautionary Tale Abstract We develop a multi-period, multi-task principal-agent model in which the principal knows the mapping from actions to outputs, but the agent does not. The agent can learn about the production function over time by exerting e↵ort and observing output. To test the model’s predictions, we conduct a field experiment in fifty Houston public schools, where students, parents, and teachers were rewarded with financial incentives for specific inputs to the education production function. The experimental data is consistent with the model’s key predictions, though other explanations are possible. Together, both the theory and experimental evidence serve as a cautionary tale about the efficacy of incentive schemes when agents do not know the production function. 1 Introduction Principal-agent models have been used to analyze problems as diverse as executive compensation, regulation, organizational design, entrepreneurship, and accounting.1 As Kenneth Arrow points out, “economic theory in recent years has recognized that [principal-agent problems] are almost universal in the economy at least as one significant component of almost all transactions” (Arrow, 1986). In the classic framework, a principal hires an agent to perform a task for her. The agent bears a private cost of taking actions. The principal does not observe the agent’s action, rather she observes a noisy measure of it (such as profits). It is this measure that is contractible, and it is assumed that the the agent’s cost of e↵ort function, both parties’ preferences, and the stochastic mapping from actions to outputs are common knowledge between principal and agent. There have been many important extensions to the basic model (e.g., multitasking and repeated contracting), but it is standard to assume that agents know how their e↵ort a↵ects output.2 In many applications, however, it is questionable whether agents know the stochastic mapping from inputs to output. Examples abound in management, finance, and other general contracting problems. In executive management, for instance, it is equivalent to assuming a CEO knows how her actions will impact the collective goals of the board of directors. This requires knowledge of the intensity of di↵ering board-member preferences and how those preferences are aggregated–a complex issue about which board-members themselves are likely better informed through repeated interactions. In education, the standard principal-agent assumptions require that students (or their teachers) know the intricacies of the education production function when econometricians with large data sets and sophisticated statistical techniques are not aware of its functional form.3 1 For classic treatments see Mirrlees 1975, Holmstrom 1979, Grossman and Hart 1983. See Beaudry (1994), Chade and Silvers (2001), Kaya (2010), and Fryer, Holden, and Lang (2012) for notable exceptions. 3 Conversely, there are many applications (e.g. computer science, engineering or manufacturing) where the standard assumption seems applicable. 2 2 To examine the implications of relaxing this assumption for the design and efficacy of incentive schemes, we begin with a simple 2x2 conceptual apparatus–two periods and two tasks–which is both a simplification and extension of the pioneering work of Holmstrom and Milgrom (1991). In each period, a risk-neutral principal o↵ers a take-it-or-leave-it linear incentive contract to an agent, who, upon accepting the contract, takes two non-verifiable actions which we label “e↵ort.” E↵ort generates a benefit to the principal and is related to an observable (and contractable) performance measure. We assume that an agent’s type augments their e↵ort in producing output: higher type agents have higher returns to e↵ort than lower type agents, all else equal. An important assumption in the model is that the principal and agent have di↵erent information about the agent’s type before the contracting phase. The principal obtains a private and perfectly revealing signal and hence knows the agents type. The agent observes no private signal, but is assumed to have prior beliefs that are normally distributed. Since the agent does not know her type but the principal does, the contract o↵ered by the principal contains information that is payo↵ relevant to the agent. We make the stark assumption that the agent does not update her prior beliefs after observing the contract. This is a non-standard assumption for which we provide two justifications. First, it seems internally inconsistent to write down a model in which an agent does not know her own ability but makes sophisticated inferences from the incentive scheme provided. Second, there is a more standard rationale for the agent not updating her beliefs. In Appendix D we show that there exists a pooling equilibrium of the signaling game between principal and agent. In that equilibrium, all types of principal o↵er the same contract and hence the (fully rational) agent does not update her prior about ability after observing the contract. We leave it to the reader as to whether the behavioral assumption or equilibrium selection argument best rationalizes our assumption that the agent does not update her beliefs when observing the incentive scheme. The model has four primary predictions. First, incentives for a given task lead to an increase in e↵ort on that task. Second, incentives for a given task lead to a decrease in 3 e↵ort on the non-incentivized task. Further, the decrease in e↵ort on the non-incentivized task can be more or less for higher-type agents relative to lower-type agents, depending on how substitutable those tasks are in the cost of e↵ort function. Our final, and perhaps most distinguishing, theoretical result concerns the persistent e↵ects of changes in incentives due to agents updating about their ability types. We show that when the agent’s true ability on a given task is sufficiently low, the learning that comes from the provision of incentives is detrimental to the principal. In the absence of incentives the agent would exert some baseline level of e↵ort due to intrinsic motivation and hence learn “little” about her ability. Providing incentives induces more e↵ort than this and hence more learning about their ability type. When agents discover that they are lower-ability than they previously believed, they exert lower e↵ort in period two for any tasks on which there is a positive incentive slope (as in the case of optimal incentives). The average impact of an incentive contract depends on the distribution of across types, among other things. To better understand these predictions in a real-world laboratory, we analyze new data from a field experiment conducted in fifty traditionally low-performing public schools in Houston, Texas during the 2010-2011 school year.4 We provided financial incentives to students, their parents, and their teachers for fifth graders in twenty-five treatment schools.5 Students received $2 per math objective mastered in Accelerated Math (AM), a software program that provides practice and assessment of leveled math objectives to complement a primary math curriculum. Students practice AM objectives independently or with assistance on paper worksheets that are scored electronically and verify mastery by taking a computerized test independently at school. Parents also received $2 for each objective their child mastered and $20 per parent-teacher conference attended to discuss their student’s 4 The original impetus of the experiment was to study the impact of aligning parent, teacher, and student incentives on student achievement. The two-year evaluation of the experiment led to puzzling findings inconsistent with existing theory. See Fryer (2012). 5 One may worry that the experiment has incentives for teachers, parents, and students whereas the model has a single agent. If parent and teacher e↵ort has a non-negative e↵ect on student e↵ort, then this is isormorphic to our single agent model with more intense incentives and analogous to the monitoring intensity principle in Milgrom and Roberts (1991). Given the lack of impact on direct outcomes in many previous experiments using financial incentives, we chose to align incentives (Angrist and Lavy 2009, Fryer 2011a). 4 math performance. Teachers earned $6 for each parent-teacher conference held and up to $10,100 in performance bonuses for student achievement on standardized tests. In total, we distributed $51,358 to 46 teachers, $430,986 to 1,821 parents, and $393,038 to 1,734 students across the 25 treatment schools. The experimental results are consistent with the predictions of the model: the good, the bad, and the ugly. Throughout the text we report Intent-to-Treat (ITT) estimates. On outcomes for which we provided direct incentives, there were very large and statistically significant treatment e↵ects. Students in treatment schools mastered 1.087 (0.031) standard deviations (hereafter ) more math objectives than control students. On average, treatment parents attended almost twice as many parent-teacher conferences as control group parents and were 7.2 percentage points more likely to report checking their child’s homework. And, perhaps most important, these behaviors translated into a 0.081 (0.025) increase in math achievement on Texas’s statewide student assessment. Now, the bad and the ugly: the impact of our incentive scheme on reading achievement (which was not incentivized) is -0.089 (0.027) – o↵setting the positive math e↵ect. And, while higher-achieving students (measured from pre-treatment test scores) seemed to gain from the experiment on nearly every dimension, lower-achieving students had significant and lasting negative treatment e↵ects. Higher-achieving students master 1.66 more objectives, have parents who attend two more parent-teacher conferences, have 0.228 higher standardized math test scores and equal reading scores relative to high-achieving students in non-treated schools. Conversely, lowerachieving students master 0.686 more objectives, have parents who attend 1.5 more parentteacher conferences, have equal math test scores and 0.163 lower reading scores. Put differently, higher-achieving students put in significant e↵ort and were rewarded for that e↵ort in math without a deleterious impact in reading. Lower-achieving students also increased e↵ort on the incentivized task, but did not increase their math scores and their reading scores decreased significantly. These data are compatible with predictions (i) through (iii) of the model. 5 Consistent with the fourth – and most stark – prediction of the model, higher-achieving students continue to do well, maintaining a positive treatment e↵ect in math and a zero e↵ect in reading, one year after the incentives are taken away. Lower-achieving students, however, exhibit large and statistically significant decreases in both math [-.223 (0.056)] and reading achievement [-.168 (0.079)] after the incentives are removed. We argue that this is most likely explained by students learning about their own ability and not decreases in intrinsic motivation. The treatment e↵ect on the latter, gleaned from survey data, is small and statistically insignificant. Our contribution is three fold. First, we extend the classic multitask principal-agent model to a multi period, multi-type, setting in which the agent does not know the production function, but can learn it over time.6 Second, we demonstrate, using data from the first experiment designed to align the incentives of parents, teachers, and students on a common performance measure, that the e↵ort substitution problem is larger for low-ability types.7 Third, we demonstrate persistent negative e↵ects on student test scores on multiple measures – a cautionary tale on the design of incentives when agents do not know the production function.8 The next section presents a multi-period, multitasking principal-agent model. Section 3 provides details of the field experiment and its implementation. Section 4 describes the 6 See Fryer, Holden, and Lang (2012) for a single task model with similar features. Beaudry (1994) also studies a setting where the principal knows the mapping from action to output but the agent does not. In his model there are two types of agent and two possible output levels. Focusing on separating perfect Bayesian equilibria he shows that high types receive a higher base wage and a lower bonus than low types. See also Chade and Silvers (2001) and Kaya (2010). Our also relates to the so-called informed principal problem in mechanisms design first analyzed by Myerson (1983) and Maskin and Tirole (1990, 1992). This large literature studies studies the equilibrium choice of mechanisms by a mechanism designer who possess private information. The key di↵erence is that our focus is on a specific environment with hidden actions after the contracting stage, rather than on characterizing the set of equilibria in very general hidden information settings. One way to see this di↵erence is that in Maskin and Tirole (1992) actions are observable and verifiable. 7 There is a growing literature on the use of financial incentives to increase student achievement in primary (Bettinger 2010, Fryer 2011a), secondary (Angrist and Lavy 2009, Fryer 2011a, Kremer, Miguel, and Thornton 2009), and postsecondary (Angrist, Lang, and Oreopoulos 2009, Oosterbeek et al. 2010) education. 8 Psychologists often warn of the potential negative e↵ects of incentives due to intrinsic motivation. Our model and data suggests a di↵erent mechanism: rational, but potentially incorrect, learning about one’s type. 6 data collected, research design, and econometric framework used in the analysis. Section 5 presents estimates of the impact of the treatment on various test score and non-test score outcomes. The final section concludes with a more speculative discussion of the implications of the model and experimental data for the design of incentive schemes. There are three online appendices. Online A provides technical proofs of the propositions detailed in Section 2, along with other mathematical details. Online Appendix B is an implementation supplement that provides details on the timing of our experimental roll-out and critical milestones reached. Online Appendix C is a data appendix that provides details on how we construct our covariates and our samples from the school district administrative files used in our analysis. 2 A Multi-period, Multitasking Model with Learning 2.1 Statement of the problem In each of two periods, a risk-neutral principal o↵ers a take-it-or-leave-it incentive contract to an agent, who, upon accepting the contract, takes two non-verifiable actions e1 and e2 . We will typically refer to these actions as e↵ort. Each action takes values in R+ , and generates a benefit on task i of ↵i ei to the principal and a performance measure mi = ↵i ei + ✏i where ✏i ⇠ N (0, 2 i) and is independent of everything else. We will sometimes refer to the level of ↵i as the “type” of the agent on task i. We assume that only the mi ’s are contractable, and the principal o↵ers a linear incentive contract of the form s + b1 m1 + b2 m2 that the agent can accept or reject. If the agent accepts she then makes her e↵ort choice(s), the performance measure is realized, and the principal pays the agent according to the contract. A key assumption of our model is that principal knows the true value of ↵1 and ↵2 , but the agent does not. Before the contract is o↵ered the agent has a prior probability probability distribution ↵i ⇠ N (↵i , µ2i ). We assume that it is common knowledge between the principal and agent that ↵ does not change over time, and the ✏i s are independent of each other and 7 i.i.d. over time. Since the agent does not know the ↵i s but the principal does, the contract o↵ered by the principal contains information that is payo↵ relevant to the agent. We make the stark assumption that the agent does not update her prior beliefs after observing the contract. This is a non-standard assumption and thus warrants some explanation. As we mentioned in the introduction there is a certain internal inconsistency to an agent who does not know her own ability but makes sophisticated inferences from the incentive scheme provided. But there is also a more standard rationale for the agent not updating. In the appendix, adapting the analysis in Fryer-Holden and Lang (2012) to this setting, we show that there exists a pooling equilibrium of the signaling game between principal and agent. In that equilibrium all types of principal o↵er the same contract and hence the (fully rational) agent does not update her prior about ability after observing the contract. We leave it to the reader as to whether the behavioral assumption or equilibrium selection argument best rationalizes our assumption that the agent does not update. We further assume that the agent has preferences that can be represented by a utility function that exhibits constant absolute risk aversion (CARA): u(x, e) = exp  ✓ 1 2 (e + e22 ) 2 1 ⌘ x e 1 e2 ◆ , where x is the monetary payment she receives. Let U be the certainty equivalent of the agent’s outside option and normalize this to zero. Notice that the parameter (which we assume to be strictly positive) measures the degree of substitutability between the tasks. Finally, we assume that the agent is myopic and unable to borrow, and we normalized the common discount factor to 1. 8 2.2 Solving the model 2.2.1 One period, symmetric information We now solve the benchmark case where the agent also knows ↵. Given the exponential utility function and normal noise standard calculation imply that the agent receives certainty equivalent CE = 2 X 1 2 (e + e22 ) 2 1 bi ↵ i e i + s i=1 2 X ⌘ e 1 e2 i=1 2 b2i 2 i. Therefore, the principal’s problem becomes max b1 ,b2 ,s,e1 ,e2 ( 2 X (↵i bi )ei i=1 subject to e1 , e2 2 argmaxẽ1 ,ẽ2 2 X bi ↵ i e i + s i=1 ( 2 X s ) bi ↵i e˜i + s i=1 1 2 (e1 + e22 ) 2 e1 e 2 1 2 ẽ1 + ẽ22 2 2 X ⌘ i=1 2 b2i 2 i e˜1 e˜2 2 X ⌘ i=1 2 b2i 2 i ) U. The first constraint is the agent’s incentive compatibility (IC) constraint, ensuring that the e↵orts that the principal designs the incentive scheme to elicit are in fact optimal for the agent. The second is the agent’s individual rationality (IR) constraint, ensuring that the agent receives at least her outside option in expectation and hence is willing to accept the contract o↵ered by the principal. The solution to the agent’s optimization problem9 is e 1 = ↵ 1 b1 e2 , e 2 = ↵ 2 b2 e1 . 9 Note that the first-order approach is valid in this setting–that is the second-order conditions for the agent’s problem are satisfied. 9 Solving simultaneously yields b1 ↵ 1 1 b ↵ 2 2 e⇤2 = 1 b2 ↵2 e⇤1 = 2 , b1 ↵1 2 (1) . (2) Substituting these and the s that makes the agent’s participation constraint binding into the principal’s objective function and taking first-order conditions for b1 and b2 yields the following unconstrained problem.10 max b1 ,b2 (✓ 1 2 ◆ ) + b2 ↵2 (1 ) 2 1 ◆2 ✓ b2 ↵2 b2 ↵ 2 + 2 1 b1 ↵1 (1 ✓ b1 ↵ 1 1 ✓ b1 ↵ 1 1 ◆2 ! b2 ↵2 2 b1 ↵1 2 ◆✓ ⌘ 2 b 2 1 2 1 b2 ↵ 2 1 + b22 b1 ↵1 2 2 2 . ) ◆ (3) Taking first-order conditions and solving simultaneously yields the equilibrium incentive slopes (b⇤1 , b⇤2 ), (↵1 )2 + (1 )⌘ 22 (↵1 )2 + ⌘( 22 + 12 ) + ⌘ 2 12 22 (1 (↵2 )2 + (1 )⌘ 12 b⇤2 = (↵2 )2 + ⌘( 12 + 22 ) + ⌘ 2 22 12 (1 b⇤1 = 2.2.2 2) , (4) . (5) 2) One period, asymmetric information The analysis proceeds as above, but now the certainty equivalent now must account for the risk imposed on the agent because of the uncertainty about ↵, and that the agent must take an expectation over ↵ when assessing her expected payment. CE = 2 X i=1 bi ↵ i e i + s 1 2 (e + e22 ) 2 1 e 1 e2 2 X ⌘ i=1 2 b2i 2 i + e2 µ2i . The principal’s problem has the same objective function, but the IR and IC constraint 10 The participation constraint must at the optimum, otherwise the fixed payment s could be reduced and improve the principal’s payo↵ with a↵ective the agent’s incentives of payo↵. 10 now take account of the uncertainty about ↵. max b1 ,b2 ,s,e1 ,e2 ( 2 X (↵i i=1 subject to e1 , e2 2 argmaxẽ1 ,ẽ2 2 X bi )ei ( 2 X s bi ↵i e˜i + s i=1 1 2 (e1 + e22 ) 2 bi ↵ i e i + s i=1 ) e 1 e2 1 2 ẽ + ẽ22 2 1 2 X i=1 ⌘ 2 b 2 i e˜1 e˜2 2 X ⌘ i=1 2 i + e2 µ2i 2 b2i 2 i + e2 µ2i U. The solution to the agent’s optimization problem is now e 1 = ↵ 1 b1 e 2 = ↵ 2 b2 1 2 2 b ⌘µ 2 1 1 1 2 2 b ⌘µ 2 2 2 e2 e1 , e1 e2 . Solving simultaneously we have e⇤1 = 2↵1 b1 e⇤2 = 2↵2 b2 2↵2 b2 + b22 ⌘µ22 b21 ⌘µ21 , 2) 2 (1 2↵1 b1 + b21 ⌘µ21 b22 ⌘µ22 2) . 2 (1 Proceeding as before, the principal’s unconstrained problem is 11 (6) (7) ) ⇢ ✓ ✓ ◆ 1 2 µ21 ( 2↵2 b2 + 2↵1 b1 + b22 ⌘µ22 + b21 ( ⌘)µ21 ) 2 max 2 b⌘ 1 b1 ,b2 2 1 2 ( 2 1) ✓ ◆ 1 2 µ22 ( 2↵1 b1 + 2↵2 b2 + b21 ⌘µ21 b22 ⌘µ22 ) 2 + b2 ⌘ 2 2 2 ( 2 1) ↵1 b1 ( 2↵2 b2 + 2↵1 b1 + b22 ⌘µ22 + b21 ( ⌘)µ21 ) ↵2 b2 ( 2↵1 b1 + 2↵2 b2 + b21 ⌘µ21 b22 ⌘µ22 ) + + 2 ( 2 1) 2 ( 2 1) ( 2↵1 b1 + 2↵2 b2 + b21 ⌘µ21 b22 ⌘µ22 ) ( 2↵2 b2 + 2↵1 b1 + b22 ⌘µ22 + b21 ( ⌘)µ21 ) + 4 ( 2 1)2 ✓ ◆◆ 1 ( 2↵1 b1 + 2↵2 b2 + b21 ⌘µ21 b22 ⌘µ22 ) 2 ( 2↵2 b2 + 2↵1 b1 + b22 ⌘µ22 + b21 ( ⌘)µ21 ) 2 + + 2 4 ( 2 1)2 4 ( 2 1)2 (↵2 b2 ) ( 2↵1 b1 + 2↵2 b2 + b21 ⌘µ21 b22 ⌘µ22 ) 2 ( 2 1) (↵1 b1 ) ( 2↵2 b2 + 2↵1 b1 + b22 ⌘µ22 + b21 ( ⌘)µ21 ) 2 ( 2 1) (8) It is unimportant–although possible–to obtain closed-form solutions for the incentive slopes as we can proceed with our analysis using monotone comparative statics on the unconstrained objective function.11 It is worth noting, however, that it is the agent’s beliefs about the ↵s, not the true values, that end up entering into the slope of the incentive scheme. The principal must satisfy the IC and IR constraints of the agent, and these depend on the agent’s beliefs. This is precisely why manipulating the agent’s beliefs can a↵ect e↵ort. 2.2.3 Agent updating Now consider the two-period problem that the principal faces. She cannot change the agent’s actions in period 1, but after period one the agent updates her belief about ↵1 and ↵2 based on the outputs her actions generated. Thus, the choice of b1 and b2 in period 1 can a↵ect the agent’s actions in period two through these beliefs. After taking actions (e11 , e12 ) (superscripts index the period) and observing outputs (m11 , m12 ) the agent’s posterior belief about her ability on task i are: 11 By using monotone methods we do not need to address the issue of whether the principal’s problem is now convex when there is asymmetric information about ↵. 12 E[↵i |m1i ] = ↵¯i ✓ 2 i µ2i + 2 i ◆ + 2(↵i ↵i )bi 2(↵ i ↵ i )b i + b2 i ⌘µ22 2) 2 (1 b2i ⌘µ21 ✓ µ2i µ2i + 2 i ◆ . (9) In forming her posterior the agent puts some weight on her prior, and some weight on first period output, which depends on her e↵ort and her true ability. This obviously bears strong similarities to the classic career concerns model of Holmstrom (1982) in terms of the way the agent updates about her ability (see also, very closely related, Dewatripont-Jewitt-Tirole (1999)). Other than the well-know role that the signal-to-noise ratio plays, notice that the agent’s posterior is increasing in the di↵erence between ↵i and her expectation of it ↵i . This will play a key role, since the principal can increase expected output by using more intense incentives in period 1. Thus she can to some degree control of surprised the agent is, although this comes at a cost, because the IR constraint must be satisfied and that depends on the agent’s subjective first-period belief of her ability. 2.3 Results To ground ideas, the first and most basic result is a version of the e↵ort substitution problem of Holmstrom and Milgrom (1991). When it becomes optimal for incentives to become larger on task i, due to a change in primitives in period 1, it is necessarily optimal for incentives to become smaller on task j 6= i. Proposition 1 Consider the one period model. Fix consider ˆj2 < 2 j. Then b⇤i ( i , ˆj2 ) < b⇤i ( i , 2 j) 2 i, 2 j and associated b⇤i ( i2 , and b⇤j ( i , ˆj2 ) > b⇤i ( i , 2 i) and 2 j ). Our next set of results speak to interventions in incentive schemes. Here we have in mind a situation where incentives on one task are increased in period one, relative to some benchmark level. A natural interpretation is that the benchmark level is the optimal level and the increase comes from some change in (exogenous) primitives in our model, such as 13 the benefit to the principal or the precision of measurement of output. The following result says that an increase in incentives on a given task in period 1: (a) increases e↵ort on that task in period 1 for all types of agent, (b) decreases e↵ort on the other task in period 1 for all types of agent, and (c) may decreases e↵ort on the other task in period 1 more or less for higher type agents. Proposition 2 Consider the one period model and let the optimal incentive intensities in period 1 be b⇤1,1 , b⇤2,1 . Now consider a shock to the environment and new optimal incentive intensities b̂1,1 , b̂2,1 is such that b̂1,1 > b⇤1,1 and b̂2,1 = b⇤2,1 . Then (a) ê1,1 > e⇤1,1 , (b) ê2,1 < e⇤2,1 , (c) the sign of @ 2 e2 @b1 @↵1 is ambiguous. If the “type” of the agents di↵ers not only according to their ability but also to their opportunity cost of time, as measured by , the degree of substitutability between tasks, then we can establish a starker result in terms of part (c) of the previous proposition. Proposition 3 Consider the one period model and let the optimal incentive intensities in period 1 be b⇤1,1 , b⇤2,1 . Now consider a shock to the environment and new optimal incentive intensities b̂1,1 , b̂2,1 is such that b̂1,1 > b⇤1,1 and b̂2,1 = b⇤2,1 and further that the degree of substitutability between tasks changes from ê2,1 = e⇤2,1 , and that for all > 0 to 0 . Then there exists a 0 such that ê1,1 > e⇤1,1 , we have ê2,1 < e⇤2,1 . In words, if the concept of ability is enriched to include a measure of opportunity cost of e↵ort in addition to productivity on tasks, then an increase in incentives on one task leads not only to the standard e↵ort substitution problem, but also a di↵erential one across ability types. In fact, it can be the case that some types exhibit no deleterious e↵ect from the e↵ort substitution problem on the second task, while other, lower-ability types do. A high value of implies that raising e↵ort on one task increases the marginal cost of e↵ort on the other task. This is consistent with a setting where there is a third numeraire task (such as leisure) and there is heterogeneity among agents about how they value the numeraire. 14 Our final results concern the persistent e↵ects of changes in incentives due to agents updating about their types. Proposition 4 Let e⇤1 (↵i ) be agent is e↵ort on task 1 in the absence of incentives (i.e. when b1 =0). Suppose b1,1 > 0 and b2,1 = 0 and further that b1,2 = b1,2 = 0. Then there exists ↵ˆ1⇤ such that for all ↵1⇤ < ↵ˆ1⇤ : e1,2 < e⇤1 . When the agent’s true ability on task 1 is sufficiently low the learning that comes from the provision of incentives is detrimental to the principal. In the absence of incentives the agent would exert some baseline level of e↵ort due to intrinsic motivation (in our model literally zero) and hence learn “little” (again, literally zero in our model) about her ability. Providing incentives induces more e↵ort than this and hence more learning about ability. When agents discover that they are lower-ability than they thought they exert lower e↵ort in period two for any tasks on which there is a positive incentive slope (as in the case of optimal incentives). The fact that there is a cuto↵ type, above which increased period-one incentives lead to an positive update and below which leads to a negative update stems from the fact that more intense incentives in period 1 lead to a Blackwell-more-informative experiment about agent ability. But Bayes’ Rule implies that the expectation of the conditional expectation of ability given period 1 output must equal the unconditional expectation. Thus, when the experiment leads to some agents updating positively about their ability, it must also lead (from an ex ante perspective) to some agents updating negatively. Suppose now that there is correlation between agents’ abilities on the two tasks in the following sense. Suppose ↵i1 and ↵i2 are drawn from distributions F1 and F2 respectively and further suppose that these draws are not statistically independent. We will refer to this as abilities being correlated. We then have Corollary 5 If abilities are correlated then there exists ↵ such that for all ↵ < ↵ : e1,2 < e⇤1 and e2,2 < e⇤2 . 15 That is, if agents have correlated abilities incentive that induce learning that have a deleterious e↵ect on one task spill over to the other task. In its most general form, if ⇢ is the correlation coefficient between ↵1⇤ and ↵2⇤ then larger values of ⇢ lead to lower e↵ort levels on the second (spillover) task. Finally, we discuss the optimal period 1 incentive scheme. We can decompose the problem into the costs an benefits of providing additional incentives subject to the IR and IC constraints in period 1. Divide the agents into to categories: “high types” and “low types”. For agent’s whose true ability is above the ↵ˆ1⇤ of Proposition 4 (“high types”), the benefit of increased incentives in period 1 is the additional e↵ort that will be exerted in period 2 as a consequence of the updated beliefs about ability. The agent’s optimal period 2 action on task 1, given her updated beliefs is e21 = b21 E[↵1 |m11 ] 1 2 b22 E[↵2 |m12 ] , and symmetrically for task 2. Let us denote the expected increase in profit from this additional e↵ort as ⇧(e21 (e11 )). Note that this includes not only the benefit of additional e↵ort in period 2, but the cost of satisfying the period 2 IR constraint. This is is a net increase in profit. The total expected R1 benefit across all high type agents is then ↵ˆ⇤ ⇧(e21 (e11 )). For “low types” this “benefit” is 1 R ↵ˆ⇤1 2 1 negative and given by 1 ⇧(e1 (e1 )). The cost of providing additional incentives in period 1 is the same for both classes of types, since they have the same prior belief in period 1. This cost is given by the additional fixed payment s11 required to satisfy the agent’s IR constraint. Denote this cost C(b11 , b12 ).12 The objective for the principal is thus (Z Z ↵ˆ⇤1 1 2 1 max ⇧(e1 (e1 )) + ⇧(e21 (e11 )) 1 b1 12 ↵ˆ⇤1 1 C(b11 , b12 ) + 2 X (↵i1 b1i )e1i i=1 The incentive slope on the second task enters because of the quadratic cost function. 16 s1 ) , (10) and analogously for b1 , both subject to the first period IC and IR constraints. The final term is the expected profit in period 1. As before, solving the first period IR constraint for s and the replacing the first period IC constraint with the agent’s FOC we have an unconstrained problem for the principal. Taking first-order conditions and solving simultaneously leads to the optimal period 1 incentive slopes. This is, of course, rather abstract, and we do not pursue closed-form solutions here as they are not relevant for our purpose. The salient point to note, however, is that the distribution of types is a crucial determinant of the optimal incentive scheme. This is because, when agents do not know their types and the principal is constrained to o↵er a single contract to all types there is a tradeo↵ between the benefits from “teaching” higher types their true ability, and a cost of teaching lower types theirs. We return to this in our concluding remarks. 3 Program Details Houston Independent School District (HISD) is the seventh largest school district in the nation with 202,773 students. Eighty-eight percent of HISD students are black or Hispanic. Roughly 80 percent of all students are eligible for free or reduced-price lunch and roughly 30 percent of students have limited English proficiency. Table 1 provides a bird’s-eye view of the demonstration project. To begin the field experiment, we followed standard protocols. First, we garnered support from the district superintendent and other key district personnel. Following their approval, a letter was sent to seventy-one elementary school principals who had the lowest math performance in the school district in the previous year. In August 2010, we met with interested principals to discuss the details of the experiment and provided a five day window for schools to opt into the randomization. Schools that signed up to participate serve as the basis for our matched-pair randomization. All randomization was done at the school level. Prior to the randomization, all teachers in the experimental group signed a non-binding commitment form vowing to use the Accelerated Math curriculum to supplement and complement their regular math instruc- 17 tion and indicating their intention to give all students a chance to master Accelerated Math objectives on a regular basis regardless of their treatment assignment.13 After treatment and control schools were chosen, treatment schools were alerted that they would participate in the incentive program. Control schools were informed that they were not chosen, but they would still receive the Accelerated Math software – just not the financial incentives.14 HISD decided that students and parents at selected schools would be automatically enrolled in the program. Parents could choose not to participate and return a signed opt-out form at any point during the school year.15 HISD also decided that students and parents were required to participate jointly: students could not participate without their parents and vice versa. Students and parents received their first incentive payments on October 20, 2010 and their last incentive payment on June 1, 2011; teachers received incentives with their regular paychecks.16 Table 2 describes di↵erences between schools that signed up to participate and other elementary schools in HISD with at least one fifth grade class across a set of covariates. Experimental schools have a higher concentration of minority students and teachers with low-value added on math scores. All other covariates are statistically similar. A. Students Students begin the program year by taking an initial diagnostic assessment to measure mastery of math concepts, after which AM creates customized practice assignments that focus specifically on areas of weakness. Teachers assign these customized practice sheets, and students are then able to print the assignments and take them home to work on (with or without their parents). Each assignment has six questions, and students must answer at least 13 This was the strongest compliance mechanism that the Harvard Institutional Review Board would allow for this experiment. Teachers whose data revealed that they were not using the program were targeted with reminders to use the curriculum to supplement and complement their normal classroom instruction. All such directives were non-binding and did not a↵ect district performance assessments or bonuses. 14 Schools varied in how they provided computer access to students (e.g. some schools had laptop carts, others had desktops in each classroom, and others had shared computer labs), but there was no known systematic variation between treatment and control. 15 Less than 1%, 2 out of 1695 parents opted out of the program. 16 In the few cases in which parents were school district employees, we paid them separately from their paycheck. 18 five questions correctly to receive credit.17 After students scan their completed assignments into AM, the assignments are graded electronically. Teachers then administer an AM test that serves as the basis for potential rewards; students are given credit for official mastery by answering at least four out of five questions correctly. Students earned $2 for every objective mastered in this way. Students who mastered 200 objectives were declared “Math Stars” and received a $100 completion bonus with a special certificate.18 B. Parents Parents of children at treatment schools earned up to $160 for attending eight parentteacher review sessions ($20/each) in which teachers presented student progress using Accelerated Math Progress Monitoring dashboards. Appendix Figure 1 provides a typical example. Parents and teachers were both required to sign and submit the student progress dashboards and submit them to their school’s Math Stars coordinator in order to receive credit. Additionally, parents earned $2 for their child’s mastery of each AM curriculum objective, so long as they attended at least one conference with their child’s teacher. This requirement also applied retroactively: if a parent first attended a conference during the final pay period, the parent would receive a lump sum of $2 for each objective mastered by their child to date. Parents were not instructed on how to help their children complete math worksheets. 17 Accelerated Math does not have a set scope and sequence that must be followed. While the adaptive assessment assigns a set of objectives for a student to work on, the student can work on these lessons in any order they choose, and teachers can assign additional objectives that were not initially assigned through the adaptive assessment. 18 Experimental estimates of AM’s treatment e↵ect on independent, nationally-normed assessments have shown no statistically significant evidence that AM enhances math achievement. Ysseldyke and Bolt (2007) randomly assign elementary and middle school classes to receive access to the Accelerated Math curriculum. They find that treatment classes do not outperform control classes in terms of math achievement on the TerraNova, a popular nationally-normed assessment. Lambert and Algozzine (2009) also randomly assign classes of students to receive access to the AM curriculum to generate causal estimates of the impact of the program on math achievement in elementary and middle school classrooms (N=36 elementary classrooms, N=46 middle school classrooms, divided evenly between treatment and control). Lambert and Algozzine do not find any statistically significant di↵erences between treatment and control students in math achievement as measured by the TerraNova assessment. Nunnery and Ross (2007) use a quasi-experimental design to compare student performance in nine Texas elementary schools and two Texas middle schools who implemented the full School Renaissance Program (including Accelerated Math) to nine comparison schools designated by the Texas Education Agency as demographically similar. Once the study’s results were adjusted to account for clustering, Nunnery and Ross’s (2007) analysis reveals no statistically significant evidence of improved math performance for elementary or middle school students. 19 C. Teachers Fifth grade math teachers at treatment schools received $6 for each academic conference held with a parent in addition to being eligible for monetary bonuses through the HISD ASPIRE program, which rewards teachers and principals for improved student achievement. Each treatment school also appointed a Math Stars coordinator responsible for collecting parent/teacher conference verification forms and organizing the distribution of student reward certificates, among other duties. Coordinators received an individual stipend of $500, which was not tied to performance. Over the length of the program the average student received $226.67 with a total of $393,038 distributed to students. The average parent received $236.68 with a total of $430,986 distributed to parents. The average teacher received $1,116.48 with a total of $51,358 distributed to teachers. Incentives payments totaled $875,382. 4 Data, Research Design, and Econometric Model A. Data We collected both administrative and survey data from treatment and control schools. The administrative data includes first and last name, birth date, address, race, gender, free lunch eligibility, behavioral incidents, attendance, special education status, limited English proficiency (LEP) status, and four measures of student achievement: TAKS math and ELA and STAAR math and reading assessments. Toward the end of the treatment year, the TAKS assessments were administered between April 12 and April 23, 2011, with a retake administered from May 23 to May 25, 2011. At the end of the follwing year, the STAAR assessments were administered from April 24 to April 25, 2012. We use administrative data from 2008-09 and 2009-10 (pre-treatment) to construct baseline controls with 2010-11and 2011-12 (post-treatment) data for outcome measures. Our main outcome variables are the direct outcomes that we provided incentives for: mastering math objectives via Accelerated Math and attending parent-teacher conferences. 20 We also examine indirect outcomes that were not directly incentivized, including TAKS math and ELA scale scores, Stanford 10 math and ELA scale scores, and several survey outcomes. We use a parsimonious set of controls to aid in precision and to correct for any potential imbalance between treatment and control. The most important controls are reading and math achievement test scores from the previous two years and their squares, which we include in all regressions. Previous years’ test scores are available for most students who were in the district in previous years (see Table 3 for exact percentages of experimental group students with valid test scores from previous years). We also include an indicator variable that takes on the value of one if a student is missing a test score from a previous year and zero otherwise. Other individual-level controls include a mutually exclusive and collectively exhaustive set of race dummies pulled from each school district’s administrative files, indicators for free lunch eligibility, special education status, and whether a student demonstrates limited English proficiency.19 Special education and LEP status are determined by HISD Special Education Services and the HISD Language Proficiency Assessment Committee. We also construct three school-level control variables: percent of student body that is black, percent Hispanic, and percent free lunch eligible. For school-level variables, we construct demographic variables for every 5th grade student in the district enrollment file in the experimental year and then take the mean value of these variables for each school. We assign each student who was present in an experimental school before October 1 to the first school they are registered with in the Accelerated Math database. Outside the experimental group, we assign each student to the first school they attend according to the HISD attendance files, since we are unable to determine exactly when they begin attending school in HISD. We construct the school-level variables based on these school assignments. 19 A student is income-eligible for free lunch if her family income is below 130 percent of the federal poverty guidelines, or categorically eligible if (1) the student’s household receives assistance under the Food Stamp Program, the Food Distribution Program on Indian Reservations (FDPIR), or the Temporary Assistance for Needy Families Program (TANF); (2) the student was enrolled in Head Start on the basis of meeting that program’s low-income criteria; (3) the student is homeless; (4) the student is a migrant child; or (5) the student is a runaway child receiving assistance from a program under the Runaway and Homeless Youth Act and is identified by the local educational liaison. 21 To supplement each district’s administrative data, we administered a survey to all parents and students in treatment and control schools.20 The data from the student survey includes information about time use, spending habits, parental involvement, attitudes toward learning, perceptions about the value of education, behavior in school, and Ryan’s (1982) Intrinsic Motivation Inventory. The parent survey includes basic demographics such as parental education and family structure as well as questions about time use, parental involvement, and expectations. To aid in survey administration, incentives were o↵ered at the teacher level for percentages of student and parent surveys completed. Teachers in treatment and control schools were eligible to receive rewards according to the number of students they taught: teachers with between 1-20 students could earn $250, while teachers with 100 or more students could earn $500 (with fifty dollar gradations in between). Teachers only received their rewards if at least ninety percent of the student surveys and at least seventy-five percent of parent surveys were completed. In all, 93.4 percent of student surveys and 82.8 percent of parent surveys were returned in treatment schools; 83.4 percent of student surveys and 63.3 percent of parents surveys were returned in control schools. These response rates are relatively high compared to response rates in similar survey administrations in urban environments (Parks et al. 2003, Guite et al. 2006, Fryer 2010). Table 3 provides descriptive statistics of all HISD 5th grade students as well as those in our experimental group, subdivided into treatment and control. The first column provides the mean, standard deviation, and number of observations for each variable used in our analysis for all HISD 5th grade students. The second column provides the mean, standard deviation, and number of observations for the same set of variables for treatment schools. The third column provides identical data for control schools. The fourth column displays the p-values from a t-test of whether treatment and control means are statistically equivalent. See Online Appendix C for details on how each variable was constructed. 20 Parent surveys were available in English and Spanish. 22 Within the experimental group, treatment and control students are fairly balanced, although treatment schools have more black students and fewer white, Asian, LEP, and gifted and talented students. Treatment schools also have lower previous year scores in TAKS math. A joint significance test yields a p-value of 0.643, suggesting that the randomization is collectively balanced along the observable dimensions we consider. To complement these data, Appendix Figure 2 shows the geographic distribution of treatment and control schools, as well as census tract poverty rates. These maps confirm that our treatment and control schools are similarly distributed across space and are more likely to be in higher poverty areas of a city. B. Research Design We use a matched-pair randomization procedure similar to those recommended by Imai et al. (2009) and Greevy et al. (2004) to partition the set of interested schools into treatment and control.21 Recall that we invited seventy-one schools to sign up for the randomization. Fifty-nine schools chose to sign up. To conserve costs, we eliminated the nine schools with the largest enrollment among the 59 eligible schools that were interested in participating, leaving 50 schools from which to construct 25 matched pairs. To increase the likelihood that our control and treatment groups were balanced on a variable that was correlated with our outcomes of interest, we used past standardized test scores to construct our matched pairs. First, we ordered the full set of 50 schools by the sum of their mean reading and math test scores in the previous year. Then we designated every two schools from this ordered list as a “matched pair” and randomly drew one member of the matched pair into the treatment group and one into the control group. C. Econometric model 21 There is an active debate on which randomization procedures have the best properties. Imbens (2011) summarizes a series of claims made in the literature and shows that both stratified randomization and matched-pairs can increase power in small samples. Simulation evidence presented in Bruhn and McKenzie (2009) supports these findings, though for large samples there is little gain from di↵erent methods of randomization over a pure single draw. Imai et al. (2009) derive properties of matched-pair cluster randomization estimators and demonstrate large efficiency gains relative to pure simple cluster randomization. 23 To estimate the causal impact of providing financial incentives on outcomes, we estimate Intent-To-Treat (ITT) e↵ects, i.e., di↵erences between treatment and control group means. Let Zs be an indicator for assignment to treatment, let Xi be a vector of baseline covariates measured at the individual level, and let Xs denote school-level variables; Xi and Xs comprise our parsimonious set of controls. Moreover, let m denote a mutually exclusive and collectively exhaustive set of matched pair indicators. The ITT e↵ect, ⇡, is estimated from the equation below: achievementi,m = ↵ + Xi + Xs + Zs ⇡ + m✓ + "i,m (11) The ITT is an average of the causal e↵ects for students in schools that were randomly selected for treatment at the beginning of the year and students in schools that signed up for treatment but were not chosen. In other words, ITT provides an estimate of the impact of being o↵ered a chance to participate in the experiment. All student mobility between schools after random assignment is ignored. We only include students who were in treatment and control schools as of October 1 in the year of treatment.22 In HISD, school began August 23, 2010; the first student payments were distributed October 20, 2010. 5 Analysis 5.1 Direct Outcomes Panels A and B of Table 4 include ITT estimates of treatment e↵ects on incentivized outcomes – AM objectives mastered and parent-teacher conferences attended. Objectives mastered are measured in units. Results with and without our parsimonious set of controls are presented in columns (1) and (2), respectively. In all cases, we include matched pair fixed e↵ects. Standard errors are in parenthesis below each estimate. To streamline the pre22 This is due to a limitation of the attendance data files provided by HISD. Accelerated Math registration data confirms students who were present in experimental schools from the beginning of treatment. Using first school attended from the HISD attendance files or October 1 school does not alter the results. 24 sentation of the experimental results, we focus the discussion in the text on the regressions which include our parsimonious set of controls. All qualitative results are the same in the regressions without controls. The impact of the financial incentive treatment is statistically significant across all of the direct outcomes we explore. The ITT estimate of the e↵ect of incentives on objectives mastered in AM is 1.087 (0.031). Treatment parents attended 1.578 (0.099) more parent conferences. Put di↵erently, our aligned incentive scheme caused a 125% increase in the number of AM objectives mastered and an 87% increase in the number of parent-teacher conferences attended in treatment versus control schools.23 In addition, we were able to calculate the price elasticity of demand for math objectives by examining the change in AM objectives mastered before and after two unexpected price shocks as seen in Figure 1. After five months of rewarding math objective mastery at a rate of $2 per objective, we (without prompt or advance warning) raised the reward for an objective mastered in AM to $4 for four weeks starting in mid-February and then from $2 to $6 for one week at the beginning of May. Treatment students responded by increasing their productivity; the rate of objective mastery increased from 2.05 objectives per week at the price of $2 per objective up to 3.52 objectives per week at $4 per objective, and 5.80 objectives per week at $6 per objective. Taken at face value, this implies a price elasticity of demand of 0.87. Taken together, the evidence on the number of objectives mastered and parent conferences attended in treatment versus control schools as well as the response to unexpected price shocks implies that our incentive scheme significantly influenced student and parent behavior. 23 The average control school actively mastered objectives during 8.16 of 9 payment periods. One school never began implementing the program and six stopped utilizing the program at some point during the year. Of these six, one ceased active use during February, four stopped during March, and one stopped during April. All twenty-five treatment schools actively mastered objectives throughout the duration of the program. 25 5.2 Indirect Outcomes In this section, we investigate a series of indirect outcomes – standardized test scores, student investment, parental involvement, attendance, and intrinsic motivation – that are correlated with the outcomes for which we provided incentives. Theoretically, due to misalignment, moral hazard, or psychological factors, the e↵ects of our incentive scheme on this set of outcomes is ambiguous. For these, and other reasons, Kerr (1975) notoriously referred to investigating impacts on indirect outcomes as “the folly of rewarding A, while hoping for B.” Still, given the correlation between outcomes such as standardized test scores and income, health, and the likelihood of incarceration, they may be more aligned with the outcomes of ultimate interest than our direct outcomes (Fryer 2011b). A. Student Test Scores Panel A of Table 4 presents estimates of the e↵ect of incentives on testing outcomes for which students were not given incentives. These outcomes include Texas’ state-mandated standardized test (TAKS). The math and ELA assessments are normalized to have a mean of zero and a standard deviation of one across the city sample. Estimates without and with our parsimonious set of controls are presented in columns (1) and (2), respectively. As before, standard errors are in parentheses below each estimate. ITT estimates reveal that treatment students outperform control students by 0.081 (.025) in TAKS math and underperform in TAKS ELA by 0.089 (.027).24 24 It may be surprising that the impact on math scores is not larger, given the increase in e↵ort on mastering math objectives that were correlated with the Texas state test. One potential explanation is that the objectives in AM are not aligned with those assessed on TAKS. Using Accelerated Math’s alignment map, we found that of the 152 objectives in the AM Texas 5th grade library, only 105 (69.1 percent) align with any Texas state math standards (TEKS). Texas state standard alignments are available at http://www.renlearn.com/fundingcenter/statestandardalignments/texas.aspx Furthermore, matching the AM curriculum to Texas Essential Knowledge and Skills (TEKS) standards in the six sections of the TAKS math assessment reveals the AM curriculum to be heavily unbalanced; 91 out of the 105 items are aligned with only 3 sections of the TAKS assessment (1, 4, and 6). The treatment e↵ect on the aligned sections is modest in size and statistically significant, 0.137 (.028). The treatment e↵ect on the remaining (non-aligned) portions of the test is small and statistically insignificant, 0.026 (.030). Not shown in tabular form. Another, non-competing, explanation is that students substituted e↵ort from another activity that was important for increasing test scores (i.e. paying attention in class) to mastering math objectives. 26 B. Student and Parent Engagement The survey results reported in Panel B of Table 4 report measures of student and parent engagement. Students were asked a variety of survey questions including “Did your parents check whether you had done your homework more this year or last year?” and “What subject do you like more, math or reading?” Parents were also asked a variety of questions including “Do you ask your 5th grade student more often about how he/she is doing in Math class or Reading class?” Answers to these questions are coded as binary measures and treatment e↵ects are reported as a percentage change. Details on variable construction from survey responses are outlined in Online Appendix C. Treatment parents were 7.2 (2.7) percentage points more likely, relative to the control mean of 31 percent, to report that they checked their student’s homework more during the treatment year than in the pre-treatment year. Moreover, the increased parental investment was skewed heavily towards math. Treatment parents were 12.2 (2.8) percentage points more likely to ask more about math than reading homework, and treated students were 11.2 (2.3) percentage points more likely to report a preference for math over reading. C. Attendance and Intrinsic Motivation The first row of Panel C in Table 4 reports results for student attendance – a proxy for e↵ort. The treatment e↵ect on attendance rates are 0.051 (0.027) higher than their control counterparts. This amounts to treatment students attending roughly one half of an extra day of school per year. One of the major criticisms of the use of incentives to boost student achievement is that the incentives may destroy a student’s “love of learning.” In other words, providing extrinsic rewards can crowd out intrinsic motivation in some situations. There is a debate in social psychology on this issue – see Cameron and Pierce (1994) for a meta-analysis. To measure the impact of our incentive experiments on intrinsic motivation, we administered the Intrinsic Motivation Inventory, developed by Ryan (1982), to students in our experimental groups.25 The instrument assesses participants’ interest/enjoyment, perceived 25 The inventory has been used in several experiments related to intrinsic motivation and self-regulation 27 competence, e↵ort, value/usefulness, pressure and tension, and perceived choice while performing a given activity. There is a subscale score for each of those six categories. We only include the interest/enjoyment subscale in our surveys, as it is considered the self-report measure of intrinsic motivation. To get an overall intrinsic motivation score, we sum the values for these statements (reversing the sign on statements where stronger responses indicate less intrinsic motivation). Only students with valid responses to all statements are included in our analysis of the overall score, as non-response may be confused with low intrinsic motivation. The final row of Table 4 provides estimates of the impact of our incentive program on the overall intrinsic motivation score of students in our experimental group. The ITT e↵ect of incentives on intrinsic motivation is almost exactly zero – 0.005 (0.06). 5.3 Heterogenous Treatment E↵ects Table 5 investigates treatment e↵ects on number of objectives mastered and state test scores for a set of predetermined subsamples – gender, race/ethnicity, previous year’s test score, and whether a student is eligible for free or reduced price lunch.26 All regressions include our parsimonious set of controls. Gender is divided into two categories and race/ethnicity is divided into five categories: non-Hispanic white, non-Hispanic black, Hispanic, non-Hispanic Asian and non-Hispanic other race. We only include a racial/ethnic category in our analysis if there are at least one hundred students from that racial/ethnic category in our experimental group; only black and Hispanic subgroups meet this criteria. Eligibility for free lunch is used as an income proxy. We also partition students into quintiles according to their baseline TAKS math scores and report treatment e↵ects for the top and bottom quintiles. The treatment e↵ect on objectives mastered is statistically larger for girls (1.159 ) than for boys (1.012 ). Hispanic students made the strongest gains on math tests. They also [e.g., Ryan, Koestner, and Deci (1991) and Deci et al. (1994)]. 26 All other outcomes are in Appendix Tables 3. 28 mastered more objectives while their parents attended fewer conferences. Students eligible for free lunch showed statistically larger and statistically significant gains on TAKS Math (0.144 ). They also lost less ground in reading; however, the inter-group di↵erences are only marginally significant in reading. The most noticeable and robust di↵erences occur when we stratify on previous year test scores. Consistent with Proposition 4 from Section 2, high-ability students gain most from the experiment, both in comparison to high-ability students in control schools or low-ability students in treatment schools. For instance, high-ability students master 1.66 (.117) more objectives, have parents who attend two more parent-teacher conferences, have 0.228 (.082) higher standardized math test scores and equal reading scores relative to high-ability students in control schools. Conversely, low-ability students master 0.686 (0.047) more objectives, but score 0.163 (0.063) lower in reading and have similar math test scores compared with low-ability students in control schools. 5.4 Post-Treatment Outcomes The treatment ended with a final payment to students in June of 2011. A full year after the experiment, we collected data on post-treatment test scores; math and reading tests for treatment and control students during late spring of their sixth grade year. Recall that in the model, low-ability and high-ability students who are induced to put forth additional e↵ort on a given task learn their type when they observe the results of their additional exertion of e↵ort and that high-ability agents have lower cost of displaced e↵ort. If agents base future e↵ort on their beliefs about their ability-type and update their beliefs in this way, the provision of incentives could lead low-ability agents to exert less e↵ort in the future, while high-ability agents increase their expected return to e↵ort uopn learning they are a high-ability agent and exert more e↵ort in the future. Table 6 examines lasting treatment e↵ects on standardized test scores and attendance in the year following treatment. Column 1 displays the treatment e↵ects that persisted one full year after all financial incentives were withdrawn for the full group of students with valid 29 2011-12 test scores. Columns 2 and 3 display the same results for the subgroups of students in the top and bottom quintiles of pre-treatment math test scores. In columns 5 and 6, we restricted our sample to treatment students only and regressed year 1 state test scores on objectives mastered (a measure of e↵ort exerted in math) and predicted the residuals for each student. These residuals capture the di↵erence between a student’s expected score on the state test (based upon e↵ort, as measured by objectives mastered) and her actual score. Students were divided into quintiles based upon the size of this residual, with students whose residual is the most negative in the bottom quintile or, “bad shock” group and students with the largest residuals in the top quintile or, “good shock” group. Columns 5 and 6 report the coefficient on a dummy for being in the top or bottom quintile in a regression of second year test scores on residual quintiles and our standard set of parsimonious controls, including two years of lagged test scores. Point estimates are relative to the median quintile, which is omitted from the regression. While post-treatment e↵ects in the full group sample are statistically insignificant in math, negative e↵ects linger in reading – the e↵ects are -0.042 (.029) in math and -0.084 (.030) in reading – the subgroups reveal stark di↵erences between higher and lower achieving students, as well as di↵erences based upon what students may have learned about their ability from their first year e↵ort and resulting test scores. The negative e↵ect on the reading scores of lower-achieving students persist, as lower-achieving treatment students score 0.168 (.079) lower than lower-achieving control students, and there are significant spillovers into math achievement, where lower-achieving treatment students are outperformed by 0.223 (.056). Conversely, higher-achieving treatment students outperform their control group peers by 0.134 (.078) in math and 0.097 (.086) in reading. While the latter result is statistically insignificant, the positive point estimates in both subjects suggests that any spillover e↵ects were positive or neutral. Furthermore, consistent with the model, we observe that students within the treatment group who experience a “bad shock” in the sense that they underperform on the on the 2010-11 state math test relative to the amount of e↵ort they exerted in AM perform far worse on their 2011-12 standardized tests than students who 30 experience “good shock” in their 2010-11 state math test scores relative to the amount of e↵ort they exerted in AM. Students who experience “bad shocks” score 0.252 (0.055) lower than students whose test scores are best predicted by their e↵ort in AM, while students who experience “good shocks” score 0.498 (0.061) higher– a stark di↵erence of 0.75 between receiving a“bad shock” versus a “good shock” in 2010-11 on students’ 2011-12 test scores. 6 Conclusion Individuals, even school children, respond to incentives. How we design those incentives to illicit desirable short and longer term responses is far less clear. We demonstrates these complexities with theory and a field experiment. Our model has two periods and two tasks. In each period, a risk-neutral principal o↵ers a take-it-or-leave-it linear incentive contract to an agent, who, upon accepting the contract, takes two non-verifiable actions which we label “e↵ort.” E↵ort generates a benefit to the principal and is related to an observable (and contractable) performance measure. We assume that an agent’s type augments their e↵ort in producing output: higher type agents have higher returns to e↵ort than lower type agents, all else equal. The key assumptions are that individuals do not know the production function (e.g. the mapping from input to output) and, upon observing a contract, do not update their beliefs about the production function.27 It yields four predictions: First, incentives for a given task lead to an increase in e↵ort on that task. Second, incentives for a given task lead to a decrease in e↵ort on the non-incentivized task. Further, the decrease in e↵ort on the non-incentivized task can be more or less for higher-type agents relative to lower-type agents, depending on how substitutable those tasks are in the cost of e↵ort function. The most distinguishing theoretical result concerns the persistent e↵ects of changes in incentives due to agents updating about their ability types. We show that when the agent’s true ability on a given task is sufficiently low, the learning that comes from the provision of incentives is detrimental to the principal. In the absence 27 As indicated in the introduction, this is a stark assumption for which we provide two justifications. 31 of incentives the agent would exert some baseline level of e↵ort due to intrinsic motivation and hence learn “little” about her ability. Providing incentives induces more e↵ort than this and hence more learning about their ability type. When agents discover that they are lower-ability than they previously believed, they exert lower e↵ort in period two for any tasks on which there is a positive incentive slope (as in the case of optimal incentives). The average impact of an incentive contract depends on the distribution of across types, among other things. To better understand these predictions in a real-world laboratory, we analyze new data from a field experiment conducted in fifty traditionally low-performing public schools in Houston, Texas during the 2010-2011 school year. We provided financial incentives to students, their parents, and their teachers for fifth graders in twenty-five treatment schools. Students received $2 per math objective mastered in Accelerated Math (AM). Parents also received $2 for each objective their child mastered and $20 per parent-teacher conference attended to discuss their student’s math performance. Teachers earned $6 for each parentteacher conference held and up to $10,100 in performance bonuses for student achievement on standardized tests. We argue that the data from the field experiment are consistent with the model. Higherachieving students master 1.66 more objectives, have parents who attend two more parentteacher conferences, have 0.228 higher standardized math test scores and equal reading scores relative to high-achieving students in non-treated schools. Conversely, lower-achieving students master 0.686 more objectives, have parents who attend 1.5 more parent-teacher conferences, have equal math test scores and 0.163 lower reading scores. Put di↵erently, higher-achieving students put in significant e↵ort and were rewarded for that e↵ort in math without a deleterious impact in reading. Lower-achieving students also increased e↵ort on the incentivized task, but did not increase their math scores and their reading scores decreased significantly. Consistent with the fourth – and most distinguishing– prediction of the model, higherachieving students continue to do well, maintaining a positive treatment e↵ect in math and a 32 zero e↵ect in reading, one year after the incentives are taken away. Lower-achieving students, however, exhibit large and statistically significant decreases in both math [-.223 (0.056)] and reading achievement [-.168 (0.079)] after the incentives are removed. We argue that this is most likely explained by students learning about their own ability and not decreases in intrinsic motivation. The treatment e↵ect on the latter, gleaned from survey data, is small and statistically insignificant. Taken together, both the theoretical model and the experimental results o↵er a strong cautionary tale on the use of financial incentives when individuals may not know the production function. 33