Multitasking, Learning, and Incentives - Online Appendix

7 Online Appendix A: Proofs of Propositions (Not For Publication) Proof of Proposition 1. From equation (8) apply the Milgrom-Shannon monotonicity theorem to conclude that the objective function has decreasing di↵erences (b⇤1 , 12 ) and hence that db⇤1 /d 12 < 0 and that it has increasing di↵erence in (b⇤1 , 22 ) and hence that db⇤1 /d 22 > 0. Proof of Proposition 2. For parts (a) and (b): from equation (6) note that de⇤1 /db1 > 0 and that de⇤2 /db1 < 0 and analogously for e⇤2 . Part (c): note that @ 2 e2 = @b1 ↵1 2 1 , (12) is of indeterminate sign. Proof of Proposition 3. Trivial. Follows directly from the intermediate value theorem. Proof of Proposition 4. Consider increasing b1,1 as stated in the proposition. Recall that the agent’s posterior belief after observing period 1 output is ✓ ✓ ◆ ◆ 2 µ2i 2(↵i ↵i )b1 2(↵ i ↵ i )b2 + b22 ⌘µ22 b21 ⌘µ21 i 1 E[↵i |mi ] = ↵¯i + . 2) µ2i + i2 2 (1 µ2i + i2 Consider two agents 1 and 2 with ↵11 > ↵21 . The di↵erence in posterior beliefs is E[↵11 |m1i ] E[↵12 |m1i ]. Since they have a common prior the term, and thus the di↵erence is ✓ ◆ µ2i 2b1 (↵11 ↵12 ) 2 b2 (↵21 ↵22 ) . 2) 2(1 µ2i + i2 Note that since ↵11 > ↵12 by construction, this has increasing di↵erences in (b1 , ↵1 ). By the definition of conditional probability it must be that E[E[↵|m1 ]] = E[↵] = ↵ > 0. Since E[↵1 |m1i ] E[↵1 |m1i ] = 0 for any ↵ and appealing to the intermediate value theorem the result is established. 8 Online Appendix B: Implementation Manual (Not For Publication) Schools We identified 71 low-performing elementary schools in the district based upon the average grade 5 scores on the Texas Assessment of Knowledge and Skills (TASKS) that would benefit from inclusion in the Math Stars incentive program. On Thursday, September 2, 2010, HISD leadership held an introductory meeting with principals and math teachers from these lowperforming elementary schools. After presenting an overview of the research design we invited them to commit to participate by signing a pledge to implement the Math Stars program with fidelity to the research design. Schools had five days to consider their commitment to the program (within a day, however, over two-thirds of the schools invited had already indicated their commitment and interest by signing a School Commitment Letter.) By Tuesday, September 7, 60 schools had elected to participate in the random selection process, and we conducted a random lottery to select the 25 treatment schools and the 25 control schools. Students HISD decided that students and parents at selected schools would be automatically enrolled in the program. Parents could choose not to participate and return a signed opt-out form at any point during the school year. HISD further decided that students and parents were required to participate jointly: students could not participate without their parents and vice versa. Software and Incentive Structure The Accelerated Math platform creates math assignments tailored to each student’s ability level, enabling students to take brief online assessments to gauge achievement in mathematics. For fifth grade, math objectives fall into the following subject areas: Number Sense and Operations; Algebra; Geometry and Measurement; and Data Analysis, Statistics, and Probability. Students began the program year by taking an initial diagnostic assessment to measure mastery of math concepts, after which AM creates customized practice assignments that focus specifically on areas of weakness. Teachers assign these custom assignments and students are then able to print the assignments and take them home to work on (with or without their parents). Each assignment has six questions, and students must answer at least five questions correctly to receive credit. Students scan their completed assignments into AM, and the assignments are graded electronically. Teachers then administer an AM test that serves as the basis for potential rewards: students are given credit for official mastery by answering at least four out of five questions correctly. Students: Students earned $2 for every objective mastered. Students who reached the 200 objectives threshold were declared Math Stars and received a $100 completion bonus and special certificate. Additional monetary incentives were introduced during the program: during the sixth pay period (mid-February to mid-March) students received $4 for every objective mastered; during the final week of the eighth pay period (the first week of May), students received $6 for every objective mastered. Parents: Parents of children at treatment schools earned up to $160 for attending eight parent-teacher review sessions ($20/each) in which teachers presented student progress using Accelerated Math Progress Monitoring dashboards. Parents and teachers were both required to sign the student progress dashboards and submit them to their schools Math Stars coordinator in order to receive credit. Additionally, parents earned $2 for their child’s mastery of each AM curriculum objective, as long as they attended at least one conference with their child’s teacher. This requirement also applied retroactively: if a parent first attended a conference during the final pay period, the parent would receive a lump sum of $2 for each objective mastered by their child to date. Parents were not instructed on how to help their children complete math worksheets. Teachers: Fifth grade math teachers at treatment schools received $6 for each academic conference held with a parent in addition to being eligible for monetary bonuses through the HISD ASPIRE program, which rewards teachers and principals for improved student achievement. Each treatment school also appointed a Math Stars coordinator responsible for collecting parent/teacher conference verification forms and printing and distributing student reward certificates, among other duties. Each coordinator received a stipend of $500, but this amount was not tied to performance. Principals: Principals at treatment schools were eligible for monetary bonuses through the HISD ASPIRE program, which rewards teachers and principals for improved student achievement. Training and Program Launch Once schools were selected, the Accelerated Math program was ordered for treatment and control schools, as well as computers and scanners for each school (depending on the number of students and classrooms). AM was installed in treatment schools on September 10 and control schools on September 20. HISD also hired a district-based program manager who was trained in using AM as well as a technology support sta↵ member. On September 10, a welcome packet in English and Spanish was sent home with students. The packet included a detailed description of the program, a program calendar, answers to frequently asked questions, and an opt-out form. Parents who decided they did not want their student(s) to participate in the incentive component of the Math Stars program were able to return a signed opt-out form at any point during the school year; however, students were not able to opt out of using the Accelerated Math platform. Meanwhile, treatment schools identified in-school coordinators within one day of being randomly selected; coordinators primary duties included collecting parent-teacher conference sheets and distributing checks and reward certificates to students on pay day. To e↵ectively train participating schools sta↵ to use the Accelerated Math program, Renaissance Learning sta↵ conducted teacher and coordinator training in treatment schools the week beginning September 13 (teachers in control schools were trained from September 28-29.) Teacher training consisted of coaching teachers in how to use the Accelerated Math platform to provide practice and assessment opportunities for students at di↵erent skill levels. To ensure di↵erentiated instruction, students were able to test within multiple grade levels of objectives. Therefore, a library or bank of Accelerated Math objectives, practice questions, and assessments – spanning second through seventh grades – were available from which teachers could pull assignments that students could master. However, starting in February – four full months after the beginning of the program – teachers were restricted from drawing objective assignments from libraries below fourth grade equivalency. After brief site visits to ensure that experimental schools’ technological infrastructures were properly in place, teachers were re-trained in how to use Star Math (a companion program to the Accelerated Math platform that was already in place in the HISD schools), which allows classroom teachers to administer a customized diagnostic test to students to assess skill levels within certain grade-level objectives. Therefore, to determine the grade level at which each student should begin their mastery of objectives, teachers began administering student diagnostic assessments the week beginning Monday, September 20. Within two days, 92 percent of students in treatment schools had taken the diagnostic. Payment Process Preparation and Set-up: At the conclusion of each pay period, the district-based program manager would begin processing student and parent payments along two fronts: first, extracting student performance data from the Accelerated Math platform, removing students who opted out, and calculating student rewards ($2/per objective mastered); second, collecting parent-teacher conference dashboards from school coordinators and inputing parent attendance figures. These two data points are consolidated in a pay file and organized by school. After all parent conference data was collected and inputed, the pay file was sent to EdLabs to complete the payment algorithm and conduct a few basic audits. The pay file was then sent back to the district program manager, who reformatted and finalized the file for the HISD finance office, who uploaded payment information to JP Morgan Chase. Checks were printed, bundled by school, and delivered to each school. EdLabs also used the pay file to create reward certificates for every student receiving a payment. The certificate detailed how many math objectives the student mastered during the last period, the cumulative total, and the current financial earnings. When students passed the 200 objective threshold, they received a special certificate in addition to their $100 bonus. Payment Logistics: School coordinators received student and parent checks and student certificates one day prior to pay day. Each school planned pay day di↵erently, but there was striking uniformity: typically a small assembly was held in the cafeteria during which checks and certificates were distributed and students were recognized for their achievements. Parents were often in attendance as well to acknowledge their children and receive their checks. Bonus Rounds The first several pay periods of Math Stars yielded high rates of participation among both students (i.e. percentage of students mastering at least one objective and receiving payment) and parents (i.e. percentage of parents attending a conference with their students teacher). As a result of smooth implementation and general enthusiasm about the program among students and sta↵members, HISD and EdLabs introduced two bonus rounds: during the entire sixth pay period, (February 14 through March 11), students received $4 (rather than the usual $2) for each objective mastered. During the final week of the eighth pay period (May 2 through May 5), students received $6 for each objective mastered. These changes were communicated to students primarily through posters hung throughout the school and flyers sent home in weekly folders. There were two primary objectives in introducing these bonus rounds: first, the additional incentive was meant to strengthen students preparation for end-of-year testing. The first ($4) bonus round took place just prior to the Texas Assessment of Knowledge and Skills (TAKS), while the second ($6) bonus round took place prior to the Stanford 10. Second, a sub-experiment was being conducted to estimate a demand curve for math objectives; i.e. asking whether a student will devote more e↵ort to mastering math objectives relative to the increase in the reward. Site Visits In an e↵ort to gather extensive qualitatitve data on the implementation of HISD’s Math Stars program, EdLabs conducted brief site visits to all 25 treatment schools. EdLabs observed classrooms, interviewed students, teachers, and school leaders, and developed, with extensive help from HISD program personnel, a site visit rubric. In addition to providing a comprehensive collection of qualitative school-level data to use in the evaluation of the Math Stars program (i.e. correlating school-level performance with observed implementation indicators), the site visits also supplied the district-based program manager with additional best practices to share with other schools during the last few pay periods of the program. 9 Online Appendix C: Variable Construction (Not For Publication) Attendance Rates When calculating the school-level attendance rate, we consider all the presences and absences for students when they are enrolled at each school. Individual attendance rates account for all presences and absences for each particular student, regardless of which school the student had enrolled in when the absence occurred. E↵ort Index To gauge how treatment a↵ected students’ e↵ort, we surveyed students about how strongly they agreed with the following six statements: (1) Students in my school are usually on time for class (2) Students in my classes usually turn in their homework (3) Students in my classes usually ask questions (4) I am satistified with what I have achieved in my classes (5) I have pushed myself to completely understand my lessons in school (6) I could do much better in school if I worked harder. In each case, students were instructed to indicate whether they believed the statement is totally untrue, mostly untrue, somewhat true, mostly true, or totally true. This responses were coded on an integer scale ranging from 1-5, with 1 corresponding to “totally untrue.” To construct our index of e↵ort, we added up the numeric values on all five responses (inverting the sign on question 6) and normalized the sum to have a mean of zero and a standard deviation of one. We only calculate an index for students with a valid response for all five statements, as nonresponse might otherwise be confused with strong disagreement. When individual questions appear as dependent variables in regressions, they were normalized similarly. Free Lunch Regressions include a dummy variable equal to one if a student is eligible for free or reducedprice lunch. Gifted and Talented HISD o↵ers two Gifted and Talented initiatives: Vanguard Magnet, which allows advanced students to attend schools with peers of similar ability, and Vanguard Neighborhood, which provides programming for gifted students in their local school. We consider a student gifted if he or she is involved in either of these programs. Motivation Index We disseminated part of the Intrinsic Motivation Inventory, developed by Ryan (1982), to students in our experimental group. The instrument contains many modules, but we limited our questions to those in the interest/enjoyment subscale in our surveys as it is considered the self-reported measure of intrinsic motivation. The interest/enjoyment subscale consists of seven statements on the survey: (1) I enjoy doing schoolwork very much; (2) doing schoolwork is fun; (3) I thought this was a boring activity; (4) doing schoolwork does not hold my attention at all; (5) I would describe doing schoolwork as very interesting; (6) I think doing schoolwork is quite enjoyable; and (7) while I am doing schoolwork, I think about how much I enjoyed it. Respondents are asked how much they agree with each of the above statements on a seven-point Likert scale ranging from “not at all true” to “very true.” To get an overall intrinsic motivation score, one adds up the values on each statement (reversing the sign on statements (3) and (4)). Only students with valid responses on each statement are included in our analysis of the overall score, as non-response may be confused with low intrinsic-motivation. When reporting results, we report e↵ects on scores normalized to have a mean of zero and a standard deviation of one. Special Education and Limited English Proficiency These statuses are determined by HISD Special Education Services and the HISD Language Proficiency Assessment Committee, respectively; they enter into our regressions as dummy variables. We do not consider students who have recently transitioned out of LEP status to be of limited English proficiency. Suspensions The school-level count of suspensions includes both in-school and out-of-school suspensions, regardless of the nature of the infraction. Race/Ethnicity We code the race variables such that the five categories – white, black, Hispanic, Asian and other – are collectively exhaustive and mutually exclusive. Hispanic ethnicity is an absorbing state. Hence “white” implies non-Hispanic white, “black” non-Hispanic black, and so on. Survey Responses Some of the indirect outcomes reported in the paper include survey responses. We include two questions from the student survey. First, students were asked “Did your parents check your homework this year more than last year?” We code responses of “more this year” as 1 and responses of either “more last year” or “about the same” as 0. Second, students were asked “What subject do you like better, math or reading?” We code responses of “math” as 1 and “reading” as 0. We also report the results of one question from the parent survey. Parents were asked “Do you ask your 5th grade student more often about how he/she is doing in math class or reading class?” We code responses of “math class” as 1 and responses of either “reading class” or “no di↵erence” as zero. Teacher Value-Added HISD officials provided us with 2009-10 value-added data for 3,883 middle and elementary school teachers. In Table 2, we present calculations based on the district-calculated Cumulative Gain Indices. We normalize these indices such that the average teacher in each subject has a score of zero and the sample standard deviation is one. These scores are then averaged within each school. Test Scores We observe results from the Texas Assessment of Knowledge and Skills (TAKS) and the Stanford 10. For ease of interpretation, we normalize raw scores to have a mean of zero and a standard deviation of one within grades, subjects, and years. Treatment Due to a limitation in the attendance data provided by HISD, we are unable to determine the dates on which students enrolled in their current schools. AM registration files provides a “snapshot” file that records each students’ enrolled school as of October 1. We include students in one of the 25 treatment schools on October 1, 2010 in our treatment group (the control group is defined similarly). Our results are not sensitive to changing the treatment assignment based on the first school attended during the 2010-11 school year. Table 1: Summary of Math Stars Houston Incentives Experiment Schools 50 (of 70 eligible) HISD schools opted in to participate, 25 schools randomly chosen for treatment. All treatment and control schools were provided complete Accelerated Mathematics software, training, and implementation materials (handouts and practice exercises). Treatment Group 1,693 5th grade students: 27.5% black, 70.1% Hispanic, 55.5% free lunch eligible Control Group 1,735 5th grade students: 25.7% black, 68.2% Hispanic, 53.6% free lunch eligible Outcomes of Interest TAKS State Assessment, STAAR State Assessment (post-treatment),Number of Math Objectives Mastered, Parent Conference Attendance, Measures of Parent Involvement, Measures of Student Motivation and E↵ort Test Dates Year 1: TAKS: April 12-23, 2011; TAKS Retake: May 23-25, 2011; Stanford 10: May 8-10, 2011 Year 2: STAAR: April 24-25, 2012 Objectives Database Students took a diagnostic test and were assigned math objectives to practice based upon their measured deficiencies. Incentive Structure Students paid $2 per objective to practice a math objective and pass a short test to ensure they mastered it. Additional Incentives $100 for mastering 200th objective (cumulatively) Frequency of Rewards Paydays were held every 3-4 weeks Operations $875,000 distributed in incentives payments, 99% consent rate. 2 dedicated project managers. Notes. Each row describes an aspect of treatment indicated in the first column. Entries are descriptions of the schools, students, outcomes of interest, testing dates, objectives database, incentive structure, additional incentives, frequency of rewards and operations. See Appendix A for more details. The numbers of treatment and control students given are for those students who have non-missing reading or math test scores. Table 2: Pre-Treatment Characteristics of Non-Experimental and Experimental Schools Non-Exp. Exp. E vs. NE 5th Grade 5th Grade p-value Treatment Control Teacher Characteristics Percent male 0.161 0.183 0.105 0.174 0.191 (0.079) (0.078) (0.074) (0.082) Percent black 0.322 0.370 0.307 0.366 0.374 (0.255) (0.292) (0.330) (0.257) Percent Hispanic 0.343 0.365 0.547 0.352 0.377 (0.213) (0.202) (0.222) (0.183) Percent white 0.290 0.222 0.033 0.236 0.207 (0.233) (0.158) (0.141) (0.176) Percent Asian 0.034 0.032 0.798 0.029 0.035 (0.039) (0.032) (0.030) (0.035) Percent other race 0.010 0.011 0.838 0.015 0.007 (0.015) (0.022) (0.026) (0.016) Mean teacher salary / 1000 51.942 52.079 0.674 52.088 52.071 (2.058) (1.848) (1.706) (2.014) Mean years teaching experience 11.878 12.082 0.657 12.222 11.942 (2.781) (2.656) (2.476) (2.870) Mean Teacher Value Added: Math 0.040 -0.162 0.031 -0.211 -0.113 (0.468) (0.586) (0.417) (0.722) Mean Teacher Value Added: Reading 0.040 -0.121 0.080 -0.128 -0.113 (0.465) (0.566) (0.411) (0.696) Student Body Characteristics # of suspensions per student 0.096 0.106 0.606 0.087 0.126 (0.096) (0.155) (0.108) (0.192) # of days suspended per student 0.214 0.261 0.365 0.225 0.297 (0.988) (0.344) (0.290) (0.395) Total Enrollment 2009-2010 727.467 593.068 0.000 606.522 579.251 (202.807) (142.169) (163.744) (117.878) Number of Schools 130 50 25 T vs. C p-value 0.317 0.777 0.417 0.668 0.315 0.224 0.523 0.326 0.456 0.779 0.883 0.925 0.718 25 Notes: This table reports school-level summary statistics for our aligned incentives experiment. The non-experimental sample includes all HISD schools with at least one 5th grade class in 2009-10. Column (3) reports p-values on the null hypthesis of equal means in the experimental and non-experimental sample. Column (6) reports the same p-value for treatment and control schools. Each test uses heteroskedasticity-robust standard errors, and the latter test controls for matched-pair fixed effects. Table 3: Student Pre-Treatment Characteristics HISD 5th Grade Treatment Control Student Characteristics Male 0.510 0.526 0.525 (0.500) (0.499) (0.500) White 0.078 0.019 0.046 (0.268) (0.138) (0.211) Black 0.248 0.275 0.257 (0.432) (0.447) (0.437) Hispanic 0.632 0.701 0.682 (0.482) (0.458) (0.466) Asian 0.030 0.001 0.009 (0.172) (0.035) (0.094) Other Race 0.012 0.003 0.006 (0.109) (0.055) (0.077) Special Education Services 0.098 0.108 0.086 (0.297) (0.311) (0.281) Limited English Proficient 0.307 0.293 0.336 (0.461) (0.455) (0.473) Gifted and Talented 0.193 0.138 0.166 (0.394) (0.345) (0.373) Economically Disadvantaged 0.828 0.929 0.909 (0.377) (0.257) (0.287) Free or Reduced Price Lunch 0.513 0.555 0.536 (0.500) (0.497) (0.499) TAKS Math 09-10 0.000 -0.142 -0.082 (1.000) (0.944) (0.954) TAKS ELA 09-10 0.000 -0.166 -0.152 (1.000) (0.934) (0.956) Missing Previous Math Scores 0.129 0.117 0.114 (0.336) (0.321) (0.317) Missing Previous ELA Scores 0.134 0.125 0.122 (0.340) (0.331) (0.327) p-value from joint F-test 0.643 Student Outcomes Participated in Program 0.000 Periods Treated Observations 0.111 (0.314) 0.944 (2.717) 0.966 (0.180) 8.473 (1.739) 0.001 (0.034) 0.003 (0.107) 15389 1693 1735 T vs. C. p-value 0.504 0.000 0.015 0.876 0.002 0.364 0.668 0.017 0.040 0.219 0.349 0.043 0.629 0.448 0.514 0.000 3428 Notes: This table reports summary statistics for our aligned incentives experiment. The sample is restricted to 5th grade students with valid test score data for the 2010 - 2011 school year. Column (4) reports p-values on the null hypothesis of equal means in treatment and control groups using heteroskedasticity-robust standard errors and controls for matched-pair fixed effects. Table 4 - Mean Effect Sizes (Intent to Treat Estimates): Indirect Outcomes Raw Controlled A. Student Achievement State Math 10-11 0.077*** 0.081*** (0.024) (0.025) 3128 3128 State ELA 10-11 -0.084*** -0.077*** (0.026) (0.027) 3108 3108 Aligned State Math 10-11 0.129*** 0.137*** (0.027) (0.028) 3090 3090 Unaligned State Math 10-11 0.023 0.026 (0.029) (0.030) 3090 3090 B. Survey Outcomes Parents check HW more 0.036 0.071*** (0.024) (0.027) 2041 2041 Student prefers Math to Reading 0.118*** 0.112*** (0.021) (0.023) 2356 2356 Parent asks about Math more than Rdg. 0.115*** 0.122*** (0.024) (0.028) 1908 1908 Conferences Attended 1.639*** 1.572*** (0.089) (0.099) 2052 2052 Objectives Mastered 0.978*** 1.087*** (0.029) (0.031) 3292 3292 C. Attendance and Motivation Attendance 2010-2011 0.045* 0.050* (0.026) (0.027) 3187 3187 Intrinsic Motivation 0.041 0.006 (0.056) (0.060) 2004 2004 Notes: This table reports ITT estimates of the effects of our aligned incentives experiment on various test scores and survey responses. Testing and attendance variables are drawn from HISD attendance files and standardized to have a mean of 0 and standard deviation of 1 among 5th graders with valid test scores. The survey responses included here are coded as zero-one variables; The effort and intrinsic motivation indices are constructed from separate survey responses; their construction is outlined in detail in the text of this paper and Online Appendix B. Raw regressions include controls for previous test scores, their squares, and matched-pair fixed effects. Controlled regressions also include controls for the gender, race, free lunch eligibility, special education status, and whether the student spoke English as second language. Standard errors are robust to heteroskedasticity. *** = significant at 1 percent level, ** = significant at 5 percent level, * = significant at 10 percent level. Whole Sample A. Incentivized Outcomes Objectives Mastered 1.087*** (0.031) 3292 B. Non-Incentivized Outcomes State Math 10-11 0.081*** (0.025) 3128 State ELA 10-11 -0.077*** (0.027) 3108 Male Table 5: Mean Effect Sizes (Intent to Treat) By Subsample Gender Race Free Lunch Female p-val Black Hispanic p-val Yes No 1.012*** (0.045) 1728 1.159*** (0.043) 1554 0.106*** (0.035) 1636 -0.067* (0.037) 1616 0.040 (0.037) 1491 -0.090** (0.039) 1491 0.017 0.183 0.678 0.816*** (0.045) 857 1.114*** (0.045) 2283 -0.002 (0.056) 828 -0.069 (0.071) 821 0.104*** (0.033) 2165 -0.076** (0.033) 2151 0.000 0.101 0.926 1.096*** (0.043) 1774 1.055*** (0.047) 1492 0.144*** (0.034) 1687 -0.033 (0.038) 1677 -0.006 (0.037) 1421 -0.122*** (0.041) 1411 p-val 0.519 0.003 0.106 Math Quintile Bottom Top 0.686*** (0.047) 694 1.660*** (0.117) 423 -0.004 (0.049) 663 -0.165*** (0.063) 659 0.228*** (0.082) 428 0.023 (0.083) 427 p-val 0.000 0.011 0.060 Notes: This table reports ITT estimates of the effects of the experiment on incentivized and non-incentivized outcomes for a variety of subsamples. All regressions follow the controlled specification described in the notes of previous tables. All test outcomes are standardized to have mean zero and standard deviation one among all HISD fifth graders. *** = significant at 1 percent level, ** = significant at 5 percent level, and * = significant at 10 percent level. Table 6: Mean Effect Sizes (Intent to Treat) on Second Year Outcomes By Subsample Full Previous Year Math Achievement Bad Sample Bottom Quintile Top Quintile p-value Shock State Math 11-12 (Post-treatment) -0.042 -0.223*** 0.134* -0.252*** (0.029) (0.056) (0.078) 0.000 (0.055) 2461 511 332 375 State Reading 11-12 (Post-treatment) -0.071** -0.170** 0.103 -0.196*** (0.029) (0.080) (0.086) 0.013 (0.056) 2458 516 336 375 Attendance 2011-2012 0.011 0.084 0.018 -0.070 (0.035) (0.091) (0.070) 0.538 (0.075) 2598 588 342 375 Good Shock 0.498*** (0.061) 230 0.156** (0.063) 230 0.040 (0.072) 230 p-value 0.000 0.000 0.147 Notes: Columns 1-3 report ITT estimates of the effects of the experiment on year 2 test scores and attendance. Columns 5 and 6 report regression coefficients from a regression of year 2 outcomes on dummies for whether a student received a large negative shock relative to his or her predicted year 1 test score (predicted by objectives mastered in Accelerated Math, a measure of effort). Students are broken into quintiles by the size their residuals from a regression of year 1 test scores on objectives mastered, and students with large negative residuals are in the bottom quintile, having received a while students with large positive residuals are in the top quintile, having received a . Coefficients in this regression are reported relative to the third quuintile, who experienced the median shock. The sample is restricted to the treatment group for this regression. All regressions follow the controlled specification described in the notes of previous tables. All test outcomes are standardized to have mean zero and standard deviation one among all HISD fifth graders. *** = significant at 1 percent level, ** = significant at 5 percent level, and * = significant at 10 percent level. 7 {$6} Objectives Mastered, per student, per week 6 5 4 {$4} Treatment 3 Control 2 1 0 1 2 3 4 5 6 7 8a 8b 9 Pay Period Figure 1: Objectives Mastered by Pay Period Notes: The vertical axis represents the average number of Accelerated Math (AM) objectives mastered by the average student per week. The horizontal axis represents each pay period. Prices in braces above individual points indicate changes in the price paid to treatment students per objective mastered in AM. If no price is indicated in braces above a point, treatment students were paid $2 per objective during that pay period. Control students were never paid at any price level. Appendix Table 1 - Mean Effect Sizes (ITT and TOT Estimates) First Stage ITT TOT A. Direct Outcomes Objectives Mastered 0.954*** 1.087*** 1.139*** (0.003) (0.031) (0.032) 3292 3292 3292 Conferences Attended 0.956*** 1.572*** 1.644*** (0.003) (0.099) (0.103) 2052 2052 2052 B. Indirect Outcomes State Math 10-11 0.957*** 0.081*** 0.085*** (0.002) (0.025) (0.026) 3128 3128 3128 State ELA 10-11 0.956*** -0.077*** -0.080*** (0.003) (0.027) (0.028) 3108 3108 3108 Aligned State Math 10-11 0.959*** 0.137*** 0.142*** (0.002) (0.028) (0.030) 3090 3090 3090 Unaligned State Math 10-11 0.959*** 0.026 0.027 (0.002) (0.030) (0.032) 3090 3090 3090 Parents check HW more 0.957*** 0.071*** 0.074*** (0.003) (0.027) (0.028) 2041 2041 2041 Student prefers Math to Reading 0.959*** 0.112*** 0.117*** (0.004) (0.023) (0.024) 2356 2356 2356 Parent asks about Math more than Rdg. 0.959*** 0.122*** 0.127*** (0.003) (0.028) (0.029) 1908 1908 1908 Attendance 10-11 0.962*** 0.050* 0.052* (0.002) (0.027) (0.028) 3187 3187 3187 Intrinsic Motivation 0.958*** 0.006 0.006 (0.005) (0.060) (0.062) 2004 2004 2004 Notes: This table reports ITT and TOT estimates of the effects of our aligned incentives experiment on a variety of outcomes. First-stage estimates report the causal effect of the experiment on the percentage of the school year each student spends in a treatment school(number of days present divided by 180), controlling for our full set of covariates. ITT estimates mirror those presented in earlier tables. Treatment-on-Treated estimates use randomized assignment to a treatment school to instrument for time spent in a treatment schools; the estimates can be interpreted as the effect of spending a full year in the treatment school for treated individuals. Standard errors are robust to heteroskedasticity. The construction of each dependent variables is described in the notes of previous tables. *** = significant at 1 percent level, ** = significant at 5 percent level, * = significant at 10 percent level. A. Student Achievement State Math 10-11 State ELA 10-11 Aligned State Math 10-11 Unaligned State Math 10-11 B. Survey Outcomes Parents check HW more Prefer Math to Reading Parents ask more about Math Conferences Attended Objectives Mastered C. Attendance and Motivation Attendance 2010-2011 Intrinsic Motivation Table A2: Mean Effect Sizes (Intent to Treat) By Subsample Gender Race Female p-val Black Hispanic p-val Yes Whole Sample Male 0.081*** (0.025) 3128 -0.077*** (0.027) 3108 0.137*** (0.028) 3090 0.026 (0.030) 3090 0.106*** (0.035) 1636 -0.067* (0.037) 1616 0.181*** (0.041) 1619 0.030 (0.042) 1619 0.040 (0.037) 1491 -0.090** (0.039) 1491 0.084** (0.040) 1470 0.007 (0.045) 1470 0.071*** (0.027) 2041 0.112*** (0.023) 2356 0.122*** (0.028) 1908 1.572*** (0.099) 2052 1.087*** (0.031) 3292 0.066 (0.041) 1008 0.104*** (0.032) 1214 0.088** (0.041) 945 1.691*** (0.148) 1018 1.012*** (0.045) 1728 0.075** (0.037) 1030 0.130*** (0.033) 1136 0.137*** (0.038) 960 1.416*** (0.136) 1030 1.159*** (0.043) 1554 0.050* (0.027) 3187 0.006 (0.060) 2004 0.056 (0.039) 1658 -0.061 (0.089) 1029 0.038 (0.037) 1528 0.048 (0.083) 969 0.183 0.678 0.086 0.695 0.859 0.565 0.366 0.159 0.017 0.736 0.359 -0.002 (0.056) 828 -0.069 (0.071) 821 0.021 (0.075) 808 -0.022 (0.081) 808 0.104*** (0.033) 2165 -0.076** (0.033) 2151 0.177*** (0.036) 2148 0.043 (0.038) 2148 0.014 (0.067) 527 0.065 (0.069) 575 0.173** (0.073) 480 1.708*** (0.248) 526 0.816*** (0.045) 857 0.092*** (0.034) 1414 0.119*** (0.029) 1656 0.118*** (0.036) 1334 1.474*** (0.130) 1424 1.114*** (0.045) 2283 0.132** (0.066) 821 -0.134 (0.147) 476 0.004 (0.031) 2233 0.016 (0.081) 1426 0.101 0.926 0.056 0.460 0.282 0.465 0.488 0.386 0.000 0.073 0.353 Free Lunch No 0.144*** (0.034) 1687 -0.033 (0.038) 1677 0.186*** (0.038) 1661 0.080* (0.041) 1661 -0.006 (0.037) 1421 -0.122*** (0.041) 1411 0.062 (0.044) 1409 -0.045 (0.046) 1409 0.042 (0.037) 1117 0.154*** (0.033) 1252 0.104*** (0.037) 1052 1.608*** (0.132) 1127 1.096*** (0.043) 1774 0.121*** (0.041) 911 0.056* (0.033) 1087 0.136*** (0.042) 843 1.592*** (0.155) 911 1.055*** (0.047) 1492 0.051 (0.036) 1707 -0.057 (0.087) 1072 0.031 (0.041) 1466 0.079 (0.088) 916 p-val 0.003 0.106 0.030 0.040 0.141 0.032 0.561 0.936 0.519 0.700 0.259 Math Quintile Bottom Top -0.004 (0.049) 663 -0.165*** (0.063) 659 -0.010 (0.090) 648 -0.032 (0.086) 648 0.228*** (0.082) 428 0.023 (0.083) 427 0.144*** (0.049) 427 0.129** (0.055) 427 0.006 (0.069) 387 0.153** (0.062) 506 0.032 (0.068) 356 1.492*** (0.234) 394 0.686*** (0.047) 694 0.184** (0.085) 271 0.082 (0.063) 299 0.214*** (0.077) 259 1.880*** (0.305) 270 1.660*** (0.117) 423 0.048 (0.066) 700 0.122 (0.156) 400 0.073 (0.066) 428 0.189 (0.202) 260 p-val 0.011 0.060 0.118 0.102 0.078 0.396 0.057 0.271 0.000 0.776 0.772 Notes: This table reports ITT estimates of the effects of the experiment on incentivized and non-incentivized outcomes for a variety of subsamples. All regressions follow the controlled specification described in the notes of previous tables. All test outcomes are standardized to have mean zero and standard deviation one among all HISD fifth graders. *** = significant at 1 percent level, ** = significant at 5 percent level, and * = significant at 10 percent level. Appendix Figure 1: Accelerated Math Progress Monitoring Dashboard Appendix Figure 2: Distribution of Treatment and Control Schools Across Houston Notes: The background color indicates the poverty rate for each census tract, with darker shades denoting higher concentrations of poverty. T’s and C’s mark treatment and control schools, respectively.