How Do Children Parse Naturalistic Input A New Methodology

How do children parse naturalistic input? A new methodology Introduction The study of child language development has benefitted enormously from the now widespread use of eye-tracking methods. Eye-tracking allows us to measure language comprehension unobtrusively as children look at objects or pictures or watch videos. One advantage of eye-tracking is that because children’s language comprehension precedes their production, these measures can offer a more complete picture of language competence than production measures alone, and not surprisingly, these methods have revealed that children understand more, at earlier ages, than previously believed. Bergelson and Swingley (2012), for example, recently demonstrated that even 6-month-olds understand many common nouns. A second advantage of eye-tracking is that it allows us to witness children’s processing of language in real-time, as it unfolds (e.g., Fernald et al., 2008; Trueswell et al., 1999). Such studies have demonstrated that children, like adults, process language incrementally and that their eye gaze is indicative of the interpretation(s) they are currently entertaining. Children’s real-time language comprehension abilities are important to study, as they predict outcomes both for typically developing children and children with or at risk for language delays or disorders (e.g., Fernald & Marchman, 2012; Venker et al., 2013). A limitation of current paradigms for studying children’s language comprehension, however, is that generally speaking they only test comprehension of controlled laboratory input. Children’s gaze is recorded as they hear pre-recorded auditory stimuli and view related visual stimuli. The advantage of this pre-recorded input is that it can be perfectly controlled with respect to content, as well as features such as prosody, speech rate, and coarticulatory cues. However, it differs from the naturalistic input children hear in daily life. It lacks, for example, the responsive features of parent speech (such as clarifications or repetitions if the child does not seem to understand, e.g., Brown & Bellugi, 1964; Shatz, 1978), but also lacks the disfluencies and pragmatically infelicitous contributions parents may make. The pre-recorded input also may not have been designed to resemble the input children actually hear in naturalistic contexts. Though many studies of child processing do include a corpus analysis of child-directed speech, or otherwise explicitly reference literature on how the stimuli relates to naturalistic input, still many studies simply use the exact stimuli used in adult sentence processing studies. This approach means that our understanding of child sentence processing is largely disconnected from what children are actually hearing. This gap is particularly problematic in studies of children, as we know that properties of the language input children receive predict aspects of their language development (e.g., Hart & Risley, 1995; Hoff, 2003; Hoff & Naigles, 2002; Huttenlocher et al., 1991; Rowe, 2008). An alternative approach to studying language development is the analysis of naturalistic corpora. One productive research direction in this area has been to record the language input parents provide in naturalistic contexts (e.g., play time at home), and to look for correlations between the quantity and quality of this input with children’s own language production or their scores on standard assessments. This research has overwhelmingly found that parental input predicts children’s language outcomes (e.g., Hart & Risley, 1995; Hoff, 2003; Hoff & Naigles, 2002; Huttenlocher et al., 1991; Rowe, 2008). But while these compelling findings are theoretically important and have spurred the development of interventions for at-risk children (e.g., Suskind, 2012), our understanding of the relationships between parental input and children’s language development is limited because these studies do not reveal whether, and how, children understand and make use of any given parental utterance; they only reveal correlations between what parents say and what children say, or children’s performance on standard assessments. For greater precision, we would want to know how children understand the particular input they receive from their own parent, and what specific features of this input predict the child’s comprehension and learning. In this paper we introduce a new paradigm that marries experimental and naturalistic methods to begin to understand how children understand their own parent’s unscripted speech. We use eye-tracking to obtain sensitive, real-time measures of children’s language processing, but of their own parent’s utterances rather than prerecorded auditory stimuli. Our overarching goal is to understand what specific aspects of naturalistic parental input best support children’s online language comprehension. This approach is made possible by advances in eye-tracking technology that allow us to study eye gaze behavior in more naturalistic settings. Currently available eye-trackers do not require children to wear anything, and tolerate considerable head movement, allowing us to study even very young infants as long as they can visually attend to a screen (e.g., Frank, Vul, & Johnson, 2009). Further, eye-tracking devices are becoming smaller, and easily used with a tablet or smartphone for maximum portability. To get real-time parsing data in a relatively naturalistic context, in the current study we use a small eye-tracking device, the Tobii X2-30, with a tablet computer. This device, which samples at 30 Hz, is only 7.2” inches long (and is considerably cheaper than the larger eye-tracker monitors made by the same company). It permits unobtrusive measures of children’s eye gaze as they look at pictures, websites, or books, and permits more natural interactions than eye-tracking on a big monitor; the parent and child can sit together comfortably. A few studies have examined children’s eye gaze in the context of naturalistic parentchild interactions. Guo and Feng (2013), for example, used two eye-trackers (on two different screens) to record parents’ speech and four- to five-year-old children’s eye gaze as they read a storybook, while Evans and Saint-Aubin (2005) used a head-mounted eye-tracker as the child sat with his or her parent. We believe our experimental setup improves on these in terms of naturalness by allowing the parent and child to sit together without the encumbrance of the headmounted eye-tracker, which may not be appropriate for younger children. More broadly, Linda Smith and her colleagues are using head-mounted cameras to record infants’ visual environments from their own perspectives, and this technique can be combined with eye-tracking to see how infants attend to those environments (e.g., Smith et al., 2014). Their goal is not, however, to track children’s online language processing, which requires a more controlled setup with some predictability as to what the child is likely to be looking at and what the language they are hearing is likely to be referring to. The closest similar paradigm to the one we present here is Brown-Schmidt and Tanenhaus’s (2008) work with adults; these authors collected data from two naïve adults in a referential communication task where the speech of both was unscripted. Their analyses reveal that despite the variability of unscripted speech, it is nevertheless possible to identify robust patterns in listeners’ comprehension of the referential expressions they hear, and that there are both similarities and differences in processing unscripted conversation as compared to prerecorded speech. For the current study, we borrow features of Brown-Schmidt and Tanenhaus’s (2008) approach, but with a new task designed for parent-child dyads. We focus our attention on a particular linguistic skill that develops rapidly in early childhood: the ability to quickly identify the appropriate referent in a visual scene when it is labeled. Although experimental studies have shown that by age 24 months, children can quickly, within 500 ms of the word’s onset, identify the labeled object in a simple scene (e.g., a picture of a ball and a picture of a shoe, side-by-side) (Fernald et al., 1998), in real-world situations parents may differ wildly in how they label objects. Instead of asking, “Where’s the ball?” the parent may say, “I see something red that looks like something Grandma gave you! Can you find it?” In both cases, the intended referent is the ball. But in the latter, the child has to understand the complex noun phrase, “something red that looks like something Grandma gave you” in order to resolve the referent. And of course, the parent’s choice of referential expression will likely depend on what else is in the visual scene, such as whether there are many different red toys. Therefore, we seek to understand what kinds of referential expressions parents use in various visual world contexts, as well as how children interpret these in real-time. We focus on 3- to 4-year-olds, as by this age children are skilled at processing referential expressions, including longer referential expressions containing modifiers, and can, at least in some contexts, use discourse-pragmatic information to guide their interpretation of the likely referents of these expressions (e.g., Arnold et al., 2007; Fernald et al., 2010). We report on a modest pilot study aimed at developing and testing a new paradigm that allows us to probe children’s processing of unscripted referential expressions provided by their own parents. Preliminary results indicate variance among parents in the choice of referential expression, as well as similarities between children’s processing of this input and the prerecorded input presented in prior literature. Overall, we have found this to be a relatively easy-touse paradigm that offers a balance of the variance inherent in naturalistic interactions and the control of a laboratory experiment. We conclude the paper with some suggestions for other research areas that might benefit from use of this approach. Methods Participants. Twelve dyads are included in this pilot sample, consisting of typicallydeveloping children (6 male) ranging in age from 3;2 to 4;8 years (mean 3;11 years) and one of their parents (10 mothers, 2 fathers). Dyads were recruited from Boston, MA and surrounding areas, and were acquiring English as their native language, hearing other languages less than 30% of the time. One additional dyad was excluded from the final analysis because of failure to cooperate. Materials. The stimuli consisted of pictures of people and everyday objects arranged in a grid (Figure 1). Each trial consisted of one grid, which was a page in a .pdf document. Dyads completed 16 trials. On each trial, one image was designated the “target,” and was the image the parent was instructed to encourage the child to point to. On half of the trials, the target was one of two objects from the same basic-level category (e.g., an umbrella with stripes, an umbrella with polka dots); we called such trials the Same condition (Figure 1B). On the other half of the trials, the target did not have a basic-level object match in the array, even if there was such a pair among the distracter images; we called this the Different condition (Figure 1C). No images were repeated across grids. Apparatus. The eye-tracking device, Tobii X2-30, adhered to a Tobii EyeMobile bracket. A Windows Surface Pro 2 tablet running Tobii Studio 3.2 was inserted into the bracket. These sat on a child’s play table in front of the child, who sat in an adult-sized chair. We propped the back of the bracket on top of a 2” binder to adjust the angle to be appropriate for child viewing. The parent sat next to the child in another adult-sized chair and wore laser goggles that blocked the particular wavelengths used by the Tobii X2-30. This permitted the parent to look at the screen easily (similarly to wearing sunglasses) but without being picked up by the eye-tracker. We also asked the parent to hold hands with the child, both to discourage the parent from pointing to the screen and the child from touching it (except to indicate a response). Procedure. The child and parent were first welcomed into our playroom, where the child played with an experimenter while the parent read the consent form. The parent and child were then seated in front of the experimental apparatus, and the rules of the game were explained. The parent first saw the display in Figure 1A, containing an array of 6 boxes containing the numbers 1 through 6. Parents were instructed that each page would display six images, one in each of the boxes, and that we would refer to the boxes by number on each trial. They were told that the goal of the game was to get their child to point to the image in the box we named. Children were also told that they would see pictures in the boxes and play a finding game with their parent, and that their job was to point to the picture their parent named as quickly as possible. A B C Figure 1. A. Numbered grid used during instructions. B. Representative grid from the Different Condition. C. Representative grid from the Same Condition. With the child seated at a distance of approximately 65 cm from the eye-tracker, as reported by Tobii Studio’s Track Status window, the child’s gaze was calibrated using Studio’s 5-point infant calibration, and recording began using the Screen Recording stimulus type. We then reopened the .pdf file containing the stimuli. An experimenter whispered a number into the parent’s ear on each trial. (We opted this low-tech method of prompting the parent to keep the casual gameplay aspect at the forefront. The experimenter was careful to remain out of the child’s earshot, and the results support our assumption that children did not know which image was the target before the parent’s utterance.) The parent responded at his or her own pace, and the trial concluded only when the child indicated a response. On each trial, the experimenter provided encouraging feedback, focusing on the dyad’s teamwork to ensure that the child did not feel tested or insecure about her performance (e.g., “You two make a great team!”, “You and Mommy are so fast!”). The experimenter advanced to the next trial by swiping to the next page of the .pdf or using a keyboard connected via Bluetooth. Results This procedure yields two types of data: parents’ referential expressions and children’s eye gaze. (All children pointed to the target on all trials, except for three trials that were skipped due to experimenter or parent error, e.g., the parent described the wrong target image. We also chose not to analyze pointing reaction times because a comparison of children’s eye gaze and pointing latencies suggested that some children point to the target shortly after fixating it, while others may take longer to point even after fixating the target for several seconds. This may be an interesting difference between children, e.g., systematically linked to temperament, processing speed, or certainty; or an idiosyncratic one. We leave this question to future research.) We first report on parents’ productions and then consider children’s eye gaze within different utterance types. Note that due to the small sample size and preliminary nature of these results, we report on patterns here without statistical analyses to provide an indication of how we intend to proceed with a full sample. Parents’ referential expressions. On each trial, we analyzed the first referential expression produced by the parent, including those produced after false starts but excluding clarifications, recasts, and additional referential expressions that we judged to be separate utterances. For example, in “This one’s a strawberry. Do you see a red strawberry?” we coded only “a strawberry.” We coded these expressions for several features including speech rate and number of morphemes. Here we report only on whether the expressions contained modifiers, and if so, whether the modifier appeared before or after the lexical noun naming the target image. For these analyses we therefore excluded trials on which the parent did not include a lexical noun that was, roughly, the target’s basic-level object category. Excluded responses include, “Daddy has one of these” (hammer), “Something you can eat” (strawberry), and “You were sharpening these this morning” (pencil). These composed only 8% of the data set. In the Different condition, parents produced sufficiently informative (as judged by the author) referential expressions 100% of the time. However, of these, 26% were over-modified, including modifiers in addition to a noun corresponding to the target’s basic-level object category (e.g., “red strawberry” though only one strawberry was present). Some of these unnecessary modifiers were color adjectives, which adults often produce (Sedivy, 2003), but others were not (e.g., “a Halloween cat,” “a funny-looking sock,” “a turtle reading a book”). In the Same condition, parents produced sufficiently informative referential expressions (again, as judged by the author) 99% of the time; on the other 1% of trials the parent did not immediately notice a potential ambiguity until after producing her first referential expression, and followed it with a repair (e.g., “A book. Oh, the open book”), but we excluded these trials because the child had typically already pointed or asked for clarification. Here, too, parents sometimes provided more details than necessary (e.g., “a funny-looking frog wearing a polka-dot tie” when “a frog wearing a tie” would have uniquely identified the referent), but this was rare. Interestingly, parents produced pre-nominal modifiers (e.g., “a striped umbrella”) about 40% of the time and postnominal modifiers (e.g., “an umbrella with stripes”) about 60% of the time. Because we expected the positioning of the modifier to affect how children restrict the domain of reference as they narrow in on the target, we focus on how this variable affects eye gaze and thus reflects their comprehension. Children’s eye gaze. We analyzed children’s latencies to look to the target, excluding trials on which there was excessive track loss (> 50%) and trials on which the child was looking at the target at the onset of the referential expression and continued to look at it for the duration of the expression. (We did not exclude trials on which children were initially looking at the target but looked away during the referential expression because they invariably returned gaze to the target before pointing to it.) In the Same condition, we first wanted to determine whether children were indeed using the referential expression to narrow in on the target. Thus we began by plotting children’s eye gaze to the target, the distractor (the basic-level category object match), and all of the other images in the display. We hypothesized that initially, at the onset of the referential expression on each trial, children would be roughly at chance in looking to the 6 objects, i.e., 17%. We then predicted that looking to the target and distractor would rise, with looking to the distractor decreasing as (on postnominal trials) the modifier information uniquely signaled a referent. Looks to all other images should remain low. Figure 2 depicts this pattern. Although looks to the distractor begin higher than looks to the other images (we have no explanation for this and assume for now that it is spurious), looks to the distractor begin to fall by about 2 seconds after the onset of the referential expression, while looks to the target rise at this time point. Note that this overall pattern is evident despite the large variability in the length and content of the referential expressions produced by parents. That looks to the other images remain low indicates that children’s gaze patterns are driven by the linguistic input they are receiving. With some confidence that children are using the referential expression online to restrict their hypotheses about which image is the target, we next examined latencies to children’s first Mean Proportion of Frames 0.6 Target Distractor 0.5 Everything Else 0.4 0.3 0.2 0.1 0.0 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 Time, ms from onset of referential expression Figure 2. Children’s looks to the target, distractor, and four other images in the Same condition within the first 3 seconds from the onset of the referential expression produced by the parent. Latency to look to target from offset of referential expression looks to the target in the Same condition, split by whether the parent’s referential expression contained a prenominal or postnominal modifier—in all examined cases these modifiers provide the crucial information that allows the child to determine which member of the basic-level object pair is the target. This time, we aligned the starting point to the offset rather than onset of the referential expression on each trial, because it is intuitively easier this way to understand the patterns given that the length of the referential expression differed each time (mostly due to content; speech rate was relatively similar across participants and trials). We excluded trials on which the child did not look to the target at all within 2 seconds of the referential expression’s offset. Subject means of latency to look to the target were 562 ms (SD = 251 ms) for prenominal referential expressions and 184 ms (SD = 104 ms) for postnominal referential expressions (see Figure 3). If this overall pattern manifests in a larger data set, it would suggest that children are faster when given the object category, allowing them to restrict the possible domain of reference to just the two objects in that category, and then modifying information that identifies which is the target, than they are at using the modifier itself to restrict the domain of reference. 800 600 400 200 0 Modifier BEFORE Noun -200 Modifier AFTER Noun Same Condition Figure 3. Mean subject latencies to look to the target in the Same condition on trials on which the parent produced a prenominal vs. postnominal modifier. We repeated this process for the Different condition, comparing trials on which parents produced no modifier at all to trials on which they produced prenominal and postnominal modifiers. Because in the Different condition the target did not have a basic-level object match in the display, no modifier was necessary to uniquely identify the target; the parent could felicitously say simply, e.g., “a tractor.” Mean subject latency from referential expression offset on trials with no modifier was 431 ms (SD = 218 ms), on trials with a prenominal modifier -31 ms (SD = 153 ms), and on trials with a postnominal modifier 37 ms (SD = 210 ms). (See Figure 4.) Thus, when parents produced a short referential expression, children required more time to identify the referent from its offset point than when the expression was longer. This is not surprising; however, it is interesting to note that modified expressions, even though these were not necessary and often contained redundant information, did not appear to hinder children’s processing. This echoes other findings with children processing pre-recorded auditory stimuli (Thorpe & Fernald, 2006; Morisseau et al., 2013). Latency to look to target from offset of referential expression 800 600 400 200 0 No Modifier Modifier BEFORE Noun Modifier AFTER Noun -200 Different Condition Figure 4. Mean subject latencies to look to the target in the Different condition on trials on which the parent produced no modifier, a prenominal modifier, or a postnominal modifier. Discussion We have reported the results of a pilot study in which we tested a new paradigm for studying children’s online processing of their own parent’s unscripted referential expressions produced in the context of a finding game. Although our results are preliminary, the foregoing discussion details the why and how of the paradigm. We conclude with some reflections on its limitations as well as some other potential uses of this experimental setup. Limitations. A strength of this paradigm is the fact that parents are not told what to say, but this very fact of course means a loss of experimental control. Tanenhaus and Brown-Schmidt (2008) have a helpful discussion of the limitations of studying online language processing in interactive situations; here we simply add that for the analyses presented here we did have to discard some trials because parents did not produce utterances within our parameters, and that this data loss is especially unfortunate when working with children because trials often have to be discarded anyway due to fussiness or uncooperativeness. On the other hand, this paradigm is also more controlled than it might be. Although playing finding games like “I Spy” is likely common in children’s experiences, the language that parents are using in this context is not necessarily indicative of their everyday speech to children in naturalistic contexts like mealtime. For our current purposes, this paradigm represents an appropriate balance of constrained context and unscripted speech, but for other purposes a more open-ended task (e.g., a narrative task as suggested below) might be better suited. Other uses. This general paradigm, of collecting eye gaze data from children as they parse their own parent’s unscripted speech, can be adapted to address many research questions. Here we have only examined parents’ productions of (and children’s comprehension of) referential expressions within the context of visual displays that we specifically designed to encourage certain choices. The images can easily be adapted to encourage production of other kinds of referential expressions (e.g., pronouns), other kinds of modifiers, particular verbargument structures, etc. Outside of this constrained game context, many other possibilities exist as well. For example, we have also developed a picture book paradigm in which parents tell children a story based on pictures, also presented as a .pdf file so that they can turn the pages at their own pace (akin to an e-book), but using the same experimental setup. This yields much less constrained input than the study described above, as parents have a less clear goal than they do in the game context. Questions about how well children follow along with their parents’ narratives, or about their attention to different aspects of the visual scene during the storytelling are appropriate here, as are questions about children’s attention to text versus pictures if an illustrated story with text is used instead of a picture book. Another set of questions one can ask using this paradigm is how much children with communication disorders that may impair their expressive speech nevertheless understand. Receptive language assessments are useful for providing normed, standardized measures of children’s comprehension, but this paradigm may be useful for testing understanding in children who are unable or unwilling to carry out their parents’ verbal instructions, or for studying children’s comprehension of more idiosyncratic ecologically valid stimuli that parents may be concerned with, such as picture-based systems. (Eye-tracking is used in some Augmentative and Alternative Communication (AAC) systems to allow users to communicate by directing their gaze, but more research is needed into the utility of these and other kinds of picture communication systems for young children, e.g., Gillespie-Smith & Fletcher-Watson, 2014.) Providing parents with feedback as to what elements of their speech their children best understand may be particularly valuable both in enhancing their understanding of the child’s level as well as helping them tailor their input accordingly. In sum, we have found this paradigm to have many strengths: it yields interesting data with individual variance, it is easy to use and the data relatively easy to analyze (at least, no more difficult than other kinds of eye-tracking data), the task is enjoyable for parent-child dyads, and the setup is easily portable. We hope our approach will inspire further research into children’s processing of naturalistic input.