Mouse Tracking Reveals Knowledge of Multiple Competing

Mouse Tracking Reveals Knowledge of Multiple Competing Referents During Cross-situational Word Learning 1. Introduction Learners typically encounter new words in complex environments teeming with possible meanings. Researchers have long assumed that one way learners deal with such referential ambiguity is by considering additional referential contexts in which the same word occurs (e.g., Fazly, Alishahi, & Stevenson, 2010; Fisher, Hall, Rakowitz, & Gleitman, 1994; Pinker, 1984; Siskind, 1996; Yu & Smith, 2007). Across situations, scene elements that are not relevant to a word’s meaning should occur less consistently than those that are central to its meaning. If learners could identify the elements that consistently co-occurred with a word across uses, then this would help them determine the word’s likely referent. Recent evidence suggests that under at least some circumstances, both children and adults can use cross-situational information to identify a word’s referent (e.g., Scott & Fisher, 2012; Smith & Yu, 2008; Yu & Smith, 2007; Yurovsky, Yu, & Smith, 2013). For instance, Yu and Smith (2007) presented adults with a series of training trials in which they saw four novel shapes and heard four made-up words. Across trials, each novel label consistently cooccurred with only one object. Following training, participants were tested on their knowledge of the words in a 4-alternative forced choice paradigm. During each test trial, they heard one novel label paired with its target referent and three distracter objects. The participants selected the target referent significantly more often than expected by chance, suggesting that they had used the crosssituational information to identify the words’ referents. Considerable questions remain regarding the mechanism that supports cross-situational word learning. One such question is how much information learners retain about the potential referents that occur with a word on a given observation. Some researchers have proposed that learners simultaneously accrue information about an entire set of potential referents for a given word (Fazly et al., 2010; Smith, Smith, & Blythe, 2011; Yu & Smith, 2007; Yurovsky, Fricker, Yu, & Smith, 2014). When learners first encounter a new word, they encode whatever referents co-occurred with that word. The next time they encounter the word, learners compare the current set of potential referents to the set previously stored in memory, adding new possibilities and updating the cooccurrence probabilities for previously encountered referents. Other researchers, however, have argued that learners are unable to track all of the candidate referents that co-occur with a word (e.g., Medina, Snedeker, Trueswell, & Gleitman, 2011; Trueswell, Medina, Hafri, & Gleitman, 2013). Instead, they suggest that when learners first encounter a word, they make a guess or conjecture about the word’s meaning. Learners retain this hypothesis and discard information about alternative referents. The next time learners encounter the word, they retrieve and evaluate their conjecture. If the hypothesized referent is present, then they strengthen and retain the hypothesis. If the hypothesized referent is absent, then the hypothesis is discarded and learners generate a new guess based on the current referential scene. Empirical attempts to test these accounts have yielded mixed results. Some studies suggest that learners do accumulate knowledge about multiple competing referents for a single word (e.g., Dautriche & Chemla, 2014; Smith et al., 2011; Yurovsky & Frank, 2015; Yurovsky et al., 2014). For instance, Yurovsky and Frank (2015) presented adult participants with trials in which they saw between two and eight novel objects and heard a novel word. Participants were asked to select the object that they thought the word referred to. On their next encounter with the word, participants saw a set of novel objects and either the referent they had previously selected (same trials) or one of the objects that they had not selected on their previous encounter with the word (switch trials). On both trial types, participants selected the repeated referent significantly more often than expected by chance. This suggests that in addition to retaining their conjecture across observations, participants also retained information about other referents that had occurred with the word. In contrast, several studies suggest participants retain only a single potential referent for a word across observations (Medina et al., 2011; Trueswell et al., 2013). For example, Trueswell et al. (2013) presented adults with a series trials in which a novel label was accompanied by two or five everyday objects. On each trial, participants selected the object that they thought the word referred to. Examination of participants’ trial-by-trial guesses revealed that when participants incorrectly guessed which referent went with a word, they performed at chance on the next encounter with that word. In contrast, when the participants had guessed correctly on the previous trial, they selected the target referent on the next encounter significantly more often than expected by chance. These results suggest that learners only remembered their previous guesses. If that previous guess was disconfirmed on the next trial, they appeared to be incapable of remembering which alternative referents had been present before. These two sets of conflicting findings are difficult to reconcile because the experiments have differed along many dimensions. These include whether the words referred to a single object token or an object category, the number of potential referents that occurred on each observation, whether those referents were presented in isolation or in a natural scene, and the interval between observations for a given word, among others (for discussion, Yurovsky & Frank, 2015; Yurovsky et al., 2014). Here, however, we focus on one feature that all of these prior studies have in common: participants’ knowledge about the potential referents for a word was inferred from their patterns of explicit guesses across trials. Although the referent that a participant selects provides one index of their knowledge, this measure might nevertheless fail to capture valuable information about the process by which that selection was made. A given referent selection could be arrived at via very different means. A participant might select the correct referent for a word because that participant had previously guessed that referent and thus confidently selects it again without considering other referents. Alternatively, the participant might consider how often each of the available referents had occurred with the word in the past and ultimately select the correct referent because it had the highest co-occurrence probability. In order to distinguish between these two possibilities, one would need to examine the participant’s decision-making process as it unfolded in real time. One way to capture this decision-making process is via mouse tracking: when adults are asked to click on the referent for a spoken word, the velocity, duration, and shape of their mouse trajectory is sensitive to real-time competition between alternative referents (e.g., Dale, Kehoe, & Spivey, 2007; Farmer, Cargill, Hindy, Dale, & Spivey, 2007; Spivey, Grosjean, & Knoblich, 2005). For instance, Spivey et al. (2005) presented participants with pairs of objects on a computer screen; prerecorded audio instructed participants to click on one of the objects. When the two objects were phonological competitors (e.g., pickle, picture), participants took longer to select the target, achieved maximum velocity later along the mouse trajectory, and exhibited more deviation toward the distractor as compared to trials in which the two objects’ names were phonologically dissimilar. Thus, participants’ mouse trajectories revealed competition between alternative referents that was not evident in their ultimate selection. In the present study, we investigated whether mouse tracking could provide new insights into the mechanism underlying cross-situational word learning. As participants select a referent for a word, do they consider alternative referents that were present on previous trials? To test this question, we devised a novel mouse-tracking version of Yu and Smith’s (2007) cross-situational word learning paradigm. Participants first viewed a series of training trials in which multiple novel labels occurred with multiple referents. Participants then viewed test trials in which they heard a single label while viewing four objects. On each test trial, participants selected the object that they thought the word referred to, and we tracked their mouse movements as they made this selection. In half of the test trials (competitor-absent trials), participants saw the target referent and three objects that had not previously occurred with the word. In the remaining test trials (competitor-present trials), one of the three non-target objects had occurred with the word in 50% of the training trials (high-probability competitor). If participants retain co-occurrence information for the set of potential referents for a word, then in the competitor-present trials they should experience online competition between the high-probability competitor and the target as they make their selection. This competition should impact their mouse trajectories in the competitor-present trials, resulting in differing patterns of motor dynamics across the two trial types. If, however, the participants simply track a single conjecture, then the frequency with which the available referents had previously occurred with the word should have no influence and mouse trajectories should not differ across trial types. 2. Method 2.1. Participants 208 undergraduate students (Mean age = 19.9, 138 females) completed the experiment for course credit. All the participants used their right hand to perform the task. 2.2. Stimuli Referents were high-resolution photos of 18 common objects. Each object was paired with a 1- or 2-syllable nonsense word. Words were constructed to be phonotactically probable in English and recorded by a female native English speaker. 2.3. Design Participants received 27 training trials and 18 test trials. On each training trial, participants saw four objects, one in each corner of the screen, and heard four labels played over the computer speaker (see Figure 1). The objects for each trial were randomly selected to satisfy three constraints: each word occurred six times with its target referent, three times with a high-probability competitor referent, and less than three times with all other objects. We randomly generated two unique sets of word-object pairs. In each test trial, participants saw four objects, one in each corner of the screen, and heard a single label. On competitor-present trials, the objects consisted of the target, the high-probability competitor, and two objects that had appeared in training but had never co-occurred with the word. On competitorabsent trials, objects consisted of the target and three objects that had occurred in training but had not co-occurred with the word. Participants saw one of two randomized test orders. The onscreen positions of the objects were randomly generated with the constraint that on competitor-present trials the target and the distractor could not be diagonally adjacent. This was done to maintain a consistent angle between the target and the competitor relative to the central starting position. Figure 1. Sample of a single learning trial. 2.4. Procedure Participants were instructed that they would see a series of objects and hear words and afterwards they would be tested on which word went with which object. Participants then viewed the training trials on a 65 cm by 45 cm computer screen. On each trial, participants saw four objects and heard four consecutively presented audio labels. The first label occurred 1s after the onset of the trial; each subsequent label occurred 1s after the previous label. Each trial lasted 12s; trials were separated by 1s of black screen. Following training, participants moved to a second identical computer in an adjoining room. Participants were told that they would see sets of objects accompanied by a single word and that after hearing each word, they should drag the green dot that appeared in the center of the screen to the object that they thought matched the word. Participants were told to make their decision as quickly and accurately as possible. At the start of each trial, the objects and the green dot appeared on screen; after 1s, a single audio label was delivered. The green dot was initially locked in place and unlocked at the offset of the label. This prevented the participants from making a selection prior to hearing the word. Once the participants made a selection by releasing the green dot over one of the objects, the trial ended. Trials were separated by 1s of black screen. While the participants were performing the task, we recorded the streaming x, y coordinates of the computer mouse (sample rate ≈ 71 Hz). 2.5. Data Preprocessing On each trial, participants’ final x, y coordinates were taken as their referent selection. To examine participants’ real-time decision making, all of the trajectories were remapped to orient the target location to the top-right corner. This was achieved by inverting the trajectories along the x-axis and y-axis. All trajectories were lined up to a common x, y starting position (0, 0), then individually normalized by resampling trajectories at 101 equally time-spaced values and computing, by means of linear interpolation, the corresponding mouse-coordinate values (separately for the x and y coordinate vectors). All data analyses were conducted with R 3.1.2 (2014) and the lme4 package (Bates, Maechler, Bolker, & Walker, 2015); all plots were created using the ggplot2 package (Wickham, 2009). All of the subsequent analysis of variance (ANOVA) models include participants as a random effect. 3. Results Figure 2. Average proportion of correct responses for the competitorpresent and competitor-absent trials. Error bars represent one standard error of the mean. The dashed line indicates chance performance. Figure 2 shows the proportion of correct target selections, separately by trial type. One sample t-tests revealed that participants selected the target significantly more often than expected by chance (.25) on both competitor- absent trials (M = .50, SD = .24), t(207) = 14.88, p < .001, d = 2.07, and competitor-present trials (M = .41, SD = .25), t(207) = 9.22, p < .001, d = 1.28. However, a paired samples t-test indicated participants were significantly more likely to select the target on competitor-absent trials than on competitor-present trials, t(207) = 5.93, p < .001, d = .37. This suggests that the presence of the high-probability competitor increased the difficulty of identifying the words’ referents. To determine whether participants experienced online competition between potential referents, we next examined participants’ mouse trajectories. In particular, we focused on trials where participants’ selected either the target or the high-probability competitor (i.e. referents that had previously co-occurred with the word)2. We then separated the trajectories into three trajectory types: competitor-absent (795 trajectories), competitor-present correct (target selected; 454 trajectories), and competitor-present incorrect (high-probability competitor selected; 275 trajectories). We next examined the participants’ reaction times (from label offset to mouse-click release). An ANOVA on participant’s reaction times with trajectory type as a within-subject factor revealed a significant main effect of trajectory type, F(2, 325) = 6.25, p = .002. Planned comparisons revealed that participants were significantly faster at selecting the target on competitor-absent trials (M = 1586 ms, SD = 879) than competitor-present trials (M = 1727 ms, SD = 962), z = -2.77, p = .015. Participants were also faster to select the target on competitorabsent trials than they were to select the high-probability competitor on competitor-present trials (M = 1802 ms, SD = 942), z = -2.91, p = .01. Within competitor-present trajectory types, the speed of target responses and highprobability competitor responses did not differ, z < 1. The fact that participants were slower on competitor-present trials than on competitor-absent trials suggests that they experienced real-time competition between the target and the high-probability competitor. To further examine this real-time competition, for each trajectory we computed the maximum deviation (MD): the largest positive x-coordinate deviation from an ideal response trajectory (i.e. a straight line between the starting position and the selected object) for each of the 101 time steps. For each participant, we calculated average MD values for each of the three trajectory types (see Figure 3). An ANOVA on the participants’ MD with trajectory type as a within-subject factor revealed a significant main effect of trajectory type, F(2, 325) = 5.41, p = .005. Planned comparisons revealed marginally smaller MD values for competitor-absent trajectories (M = 64.74, SD = 66.45) than competitor-present correct trajectories (M = 74.96, SD = 76.34), z = -2.29, p = .057. Competitor-absent trajectories also exhibited significantly smaller MD values than competitor-present incorrect trajectories (M = 81.65, SD = 83.44), z 2 We did not analyze trials in which participants selected one of the other distracters because the angle between the starting position and the object varied across trajectories, depending on the object selected. = -2.96, p = .008. The MD values of competitor-present correct and competitorpresent incorrect trajectories did not differ, z < 1. Participants’ tendency to deviate more in competitor-present trials suggests consideration of multiple referential alternatives. Maximum Deviation 75 50 25 0 Competitor−Absent Competitor−Present Correct Competitor−Present Incorrect Figure 3. Mean maximum deviation (MD) values separated by trajectory type. Error bars represent one standard error of the mean. Finally, angle information and sample entropy were computed using the mousetrack R package (Coco & Duran, 2015). Angle information has been used in previous mouse-tracking studies (see Dale et al., 2007) to investigate how initial movements deviated from the point of origin. Angle trajectory of mouse movements is computed as the angle relative to the y-axis for each sample in a trajectory. This provides a single measure that integrates information about xaxis and y-axis movements. A trajectory starting at the origin and moving directly to the participant’s final selection would have a constant angle trajectory (45° in our case). If participants experienced competition between referents, then this competition should be evident as more complex angle trajectories. To measure the complexity of angle trajectories, we submitted angle trajectory to an analysis of sample entropy for each trial (Richman & Mooreman, 2000). Sample entropy is an analysis that measures the complexity of a given time series. It is robust for small time series (Yentes et al., 2012) and has been used to measure the complexity or “disorder” of mouse movement trajectories (Dale et al., 2007; McKinstry, Dale, & Spivey, 2008). Sample entropy is computed for the angle trajectories by counting the number of similar sequences, m and m+1 (up to m=5), within a similarity tolerance parameter, 0.2*SDangle trajectory and then taking the negative logarithm of the ratio of similar sequence pair across m and m + 1, –ln(m/m+1). A time series of similar distances between data points across sequence lengths will result in lower sample entropy values. Larger sample entropy values are considered to have higher complexity. 0.16 Angle Entropy 0.12 0.08 0.04 0.00 Competitor−Absent Competitor−Present Correct Competitor−Present Incorrect Figure 4. Average level of angle entropy, separately by trajectory type. Error bars represent one standard error of the mean. We interpret higher values of sample entropy of angle trajectories as exhibiting competition effects through more disordered movements toward the selected object. Conversely, we interpret lower values of sample entropy as having more ordered, regular movements toward the selected object, i.e., similar angle deviations relative to the y-axis across the trajectory. An ANOVA on sample entropy (see Figure 4) with trajectory type as a within-subjects factor revealed a significant main effect of trajectory type, F(2, 325) = 6.37, p = .002. Planned comparisons revealed that trajectories were significantly less complex for competitor-absent trajectories (M = .13, SD = .06) than for competitorpresent correct trajectories (M = .14, SD = .07), z = -2.59, p = .026 and competitor-present incorrect trajectories (M = .15, SD = .08), z = -3.14, p = .005. Within the competitor-present trials, the complexity of the trajectories did not differ, z < 1. Participants exhibited an increased back-and-forth pattern when the high-probability competitor was present, regardless of whether they ultimately selected it. Thus, trajectories on competitor-present trials exhibited greater deflection and complexity than competitor-absent trials. This suggests that both the target and high-probability competitor were partially active as potential response alternatives as participants were making their selection. 4. Discussion Recent studies suggest that adults and children are able to use crosssituational information to identify the referents of novel nouns under at least some circumstances (e.g., Yu & Smith, 2007). However, it remains unclear how much information learners retain about the potential referents for a given word. The present study attempted to shed light on this question using a novel mousetracking paradigm. Adult participants were exposed to novel words in a series of ambiguous learning trials and then tested on their knowledge of the words’ referents. In some test trials, participants saw the word’s target referent and three alternative referents that had never co-occurred with the word before, while in other trials the target referent was accompanied by high-probability competitor that had repeatedly occurred with the word during training. Participants were faster and more accurate when the high-probability competitor was absent than they were when it was present. Moreover, examination of participants’ mouse trajectories revealed differing patterns of motor dynamics across the two types of test trials: when the high-probability competitor was present, participants deviated more from a straight line and followed a more complex path to the selected referent. These results are inconsistent with what one would expect if learners retained only a single conjecture about a word’s meaning. If participants only recalled their prior guess for a given word, then when that hypothesized referent was present in the test trial, they should have selected it. When that conjecture was not present in the test trial, participants should have selected a referent at random from the available choices. In either case, the decision-making process should not have been affected by how frequently the available referents had previously co-occurred with the word. Contrary to this prediction, the speed and shape of participants’ response trajectories differed across trial types, suggesting that participants were sensitive to the fact that both high-probability competitor and target previously co-occurred with the word. Thus, our results suggest that learners can accrue information about multiple potential referents for a word. Of course, this does not necessarily imply that learners retain perfect co-occurrence statistics for all potential referents for a word, nor that they are able to track multiple potential referents in all situations. The amount of co-occurrence information that learners are able to retain likely depends on a variety of factors, including how many referents present on a given observation (Smith et al., 2011; Yurovsky & Frank, 2015) and whether those referents occur in semantically coherent or themed referential contexts (Dautriche & Chemla, 2014). Our results simply demonstrate that under some conditions, learners are capable of tracking at least two potential referents that frequently co-occur with a word. More generally, our results demonstrate that continuous measures can provide information not evident in discrete guesses. For instance, our forcedchoice measure revealed that participants’ were more accurate on competitorabsent than on competitor-present trials. This finding could reflect the fact that participants were tracking multiple potential referents for each word and the resulting competition between the target and high-probability competitor increased the difficulty of the competitor-present trials. However, one could also offer a conjecture-based explanation for this same finding. Unlike competitorabsent trials, competitor-present trials included two referents that had previously co-occurred with the word. These trials therefore afforded the opportunity to confirm an incorrect conjecture: if participants previously guessed that the word referred to the high-probability competitor, they would select it if present, resulting in lower accuracy on competitor-present trials. Examining participants’ mouse trajectories as they made their guesses allowed us to tease apart these two possibilities: the differing patterns of motor dynamics across the two trial types indicated that participants experienced competition between the high-probability competitor and the target. Even when participants ultimately selected the target, the way in which they did so differed when the high-probability competitor was present. These results thus suggest that assessing the decision-making process in real-time reveals information not captured by forced-choice measures. Converging evidence for this conclusion comes from Trueswell et al. (2013), who eye-tracked participants as they performed their cross-situational learning task. Recall that Trueswell et al. (2013) found that when participants incorrectly guessed the referent for a word, they performed at chance on their next encounter with that word, suggesting that they retained only their previous conjecture. In contrast to their forced-choice responses, participants’ eye movements suggested that under some conditions, they retained knowledge of multiple referents. Specifically, when participants saw only two referents on each trial, they looked significantly longer at the target than the competitor referent, regardless of whether they had guessed correctly on their previous encounter with a word. Together with our findings, these results suggest that continuous measures have the potential to capture fine-grained information that learners retain about alternative referents, even when this information does not appear to impact their overt guesses. Our results thus suggest that mouse tracking offers a promising avenue for exploring the mechanisms behind cross-situational word learning. Future studies could potentially incorporate mouse tracking into cross-situational paradigms in which participants select a referent on each exposure to a word (e.g., Smith et al., 2010; Trueswell et al., 2013). Combining mouse tracking with repeated testing could provide new insight into the amount of fine-grained information participants retain on a given exposure as well as how this information changes across encounters. Finally, recent work suggests that when learners receive similar cross-situational evidence for two potential referents for a word, this can disrupt cross-situational learning (e.g., Bunce & Scott, in press; Yurovsky et al., 2013). Mouse tracking could be used to examine the influence of carefully controlled co-occurring distracters in order to better understand when and how competition between referents leads to breakdowns in cross-situational word learning.