Ecological Validity

Ecological Validity and “Ecological Validity” 1 Abstract Egon Brunswik coined the term ecological validity to refer to the correlation between perceptual cues and the states and traits of a stimulus. Martin Orne adapted the term to refer to the generalization of experimental findings to the real world outside the laboratory. Both are legitimate uses of the term, because the ecological validity of the cues in an experiment determines the ecological validity of the experiment itself. 2 Ecological Validity and “Ecological Validity” Recently in this journal, Shamay-Tsoory and Mendelsohn (2019) called attention to the “limited ecological validity that characterizes the bulk of the paradigms and settings” (p. 841) in cognitive neuroscience. Holleman et al. (2020), while supporting their call for more lifelike stimuli and tasks in brain-imaging studies, criticized them for misusing the concept of “ecological validity”, a term coined by Egon Brunswik (e.g., 1955, 1956). They point out, correctly, that Brunswik referred to the ecological validity of cues – i.e., the extent to which information available in the environment provides valid information about a distal stimulus – rather than the ecological validity of experiments – i.e., the extent to which experimental findings can generalize to the “real world” situation which a researcher wishes to understand (see also Holleman, Hooge, Kemner, & Hessels, 2020b). Hammond (1998), himself a former student of Brunswik, likewise referred to the “thoroughly corrupted… flagrant misuse” (para. 2) of “ecological validity” in an unpublished article circulated widely in samizdat (see also Dunlosky, Bottiroli, & Hartwig, 2009; Hammond & Stewart, 2001; Schmuckler, 2001). The alleged misuse of the term ecological validity has its origins in a paper by Martin T. Orne (1962), originally delivered at a symposium “On the Social Psychology of the Psychological Experiment” at the 1961 convention of the American Psychological Association. In that paper, Orne proposed that the essence of the experimental method is to reproduce elements of complex natural situations in the laboratory, in order to determine cause-and-effect relations. But he also argued that artificial laboratory situations have features of their own that may differ from those found in the natural environment, and which threaten generalization from the laboratory back to the real 3 world. Orne also argued that psychologists err in treating human subjects as passive responders to experimental stimuli. Instead, human subjects are sentient beings who are trying to understand the situation they are in, whether in the laboratory or the real world. To this end, they pick up on the demand characteristics of the experimental situation which he defined as “the totality of cues which convey an experimental hypothesis to the subject” (1962, p. 779). The term itself came from Aufforderungscharaktere (Lewin, 1935, p. 51; (Koffka, 1935, p. 345) -- same word from which Gibson (1966) derived his concept of affordances, The subject’s understanding of the experiment might differ from that intended by the experimenter, in which case a laboratory experiment would lack ecological validity, which Orne defined as “appropriate generalization from the laboratory to nonexperimental situations” (1962, p.776; see also Orne, 1969, 1970a, 1970b, 1973, 1981). Trained as a social psychologist as well as a psychiatrist, Orne was especially interested in the ecological validity of experiments in social psychology – particularly those involving deception (Orne & Holland, 1968; for a response, see Milgram, 1972; see also Mohl, 2017, June 30). But ecological validity is not just for social psychologists anymore. Almost 100 years after Ebbinghaus (1885/1964), for example, Neisser (1978) notoriously suggested that studies employing nonsense syllables and other verballearning paradigm slacked ecological validity, and had taught us virtually nothing about memory in real life (for responses, see Banaji & Crowder, 1989; Dunlosky et al., 2009; Kihlstrom, 1996). Clinical neuropsychologists have long worried that elderly subjects’ performance on laboratory tests does not correlate well with subjective complaints of memory function (e.g., Perlmutter, 1978; Sbordone, 2008). The real-life significance of 4 performance on laboratory tasks has also been of concern to developmental psychologists (e.g., Bronfenbrenner, 1977) – especially those concerned with developmental disabilities (e.g., Brooks & Baumeister, I977; Ford & Gaylord-Ross, 1991). Forensic psychologists, not to mention judges and juries, are often concerned with the generalization of laboratory findings to the courtroom (e.g., Goodman & Hahn, 1987; Rose, 2017). As the popularity of the concept grew, the “ecological validity” of an experiment came to be equated with its resemblance to real life. This was never Orne’s intention. Orne’s paper, with its emphasis on the individual subject’s perception of the situation, foreshadowed the cognitive revolution in social psychology. But it also precipitated social psychology’s first great crisis: a debate over whether social psychologists should do laboratory experiments at all or rather confine themselves instead to descriptive and correlational studies situated in the real world outside the laboratory (e.g., Gergen, 1973; Harre & Secord, 1972). Many experimental social psychologists cried foul, fearing that such criticisms would discredit the entire discipline (e.g., Berkowitz & Donnerstein, 1982; Sharpe & Whelton, 2016). This turn of events distressed Orne greatly: He never intended his ideas about ecological validity and demand characteristics to be used in a reflexive, nihilistic fashion. As he noted in a later commentary, “Unfortunately, this work has also been used as the basis for criticizing all experimental research in psychology and as an argument to abandon such efforts. I cannot share this view since I would not have been concerned about analyzing the nature of the psychological experiment if it were not an essential tool to elucidate psychological processes” (Orne, 1979, p. 26). 5 Far from being a misuse of the term, Orne’s notion of ecological validity is thoroughly grounded in Brunswik’s (1955, 1956) “lens model” for perception. Brunswik argued that the individual perceives the world through a "lens" of imperfect cues, as depicted in Holleman et al.’s (2020a) Figure 1 (reproduced here). The observer’s goal is to form an accurate perception of a distal stimulus, where accuracy is defined in terms of the match between the actual nature of the stimulus (size, distance, form, etc.) and the subject’s perception of it. Brunswik's lens model was subsequently generalized to the realm of judgment and decisionmaking, which is where it has been most frequently applied (e.g., Hastie & Dawes, 2010). There is an object about which a judgment is to be made: the validity of the judgment depends on the match between the features displayed by the target of the judgment, and those considered by the judge. Accuracy, whether in perception or judgment, reflects the degree of correspondence between the object and the inference. <<<<>>>> Figure 1. Schematic diagram of the lens model, after Brunswik (1955, Figure 8), reproduced from Holleman et al. (2020a) by permission. 6 On the “stimulus” side of the lens model, the distal stimulus presents various proximal cues which reflect the state of the target. These cues vary in terms of their ecological validity, or the strength of their association with the stimulus. The cues themselves have objective values, which can be thought of as the correlation between each cue and the inferred state. On the “subject” side of the model is the subset of proximal cues actually available to the perceiver. These cues have a subjective value in terms of the subject’s beliefs about their relevance to the judgment they are trying to make. The cues vary in their cue utilization, or the extent to which they play a role in the subject's judgment. Cue utilization can be thought of as the correlation between the cue and the judgment. Brunswik’s “lens”, then, serves as a kind of filter between the cues presented by the target and those utilized by the subject. If the cues utilized by the subject match the ecologically valid cues presented by the target, then the percept will be accurate or the judgment will be valid. But if there is a discrepancy between the cues presented by the target and the way they are processed by the subject, then the percept will be inaccurate, and the judgment invalid. For example, ecologically valid cues may be unavailable for some reason. Other opportunities for error arise from the subject’s judgment policy. For example, subjects may fail to utilize ecologically valid cues that are available to them; they may utilize cues that are not ecologically valid; or they may over- or undervalue cues, relative to their actual ecological validity. Traditionally, Brunswik’s lens model applies to one target and one perceiver (or judge). If subjects attend to ecologically valid cues, then their percepts and judgments will be accurate. When there is a single judge and two or more targets, the judgments 7 of the various targets will be similar, provided that the cues provided by the targets, and utilized by the subject, are also similar. When there are two or more judges of a single target, they may make different judgments depending on the cues available to them, or their judgment policies concerning cue utilization – which cues they pay attention to, how they weigh them, and the like. Orne understood that experimenters and subjects may have different perceptions of the experimental situation; in order to understand the subject’s behavior in an experiment, the experimenter must understand the subject’s point of view. This last point brings us to the matter of the ecological validity of experiments – that is, whether we can generalize from the results of an experiment to the real-world situation that inspired it. In this case there are two different targets, the real-world situation and the laboratory analogue. To the extent that the experiment and the realworld situation present the same ecologically valid cues to the subject, we can say that the experiment itself is ecologically valid. If the cue information presented by the experimental situation differs significantly from that presented by the real-world situation, however, the cues available in the experimental situation will lack ecological validity with respect to the real-world situation. Whatever behavior occurs in the laboratory will not generalize to the real-world situation that the laboratory situation is intended to model. Even an experiment containing very “real-life complex, dynamic, naturalistic stimuli” (Shamay-Tsoory & Mendelsohn, 2019, p. 851) can lack ecological validity, if the laboratory environment contains cues that are not representative of the real-world situation that inspired the experiment. 8 This is the case in some visual illusions. In the Ames Room (Ittelson, 1952), for example, the experimenter has altered the geometry of the room – the lengths of walls, the shape of the windows, the slant of the ceiling, to make it appear to be rectangular when in fact it is trapezoidal, and objects in it as different in size when in fact they are identical (Epstein, 1962). The visual cues available to and utilized by observers are ecologically valid for a normal, rectangular room, but are completely misleading when it comes to perceiving the Ames Room as it really is. This is also the case in some socialpsychological experiments involving deception. In Milgram’s (1963) experiments on obedience to authority, there are cues that suggest that the experimental situation is not what it appears to be. For example, there is nothing the “teacher” does in that experiment that the experimenter could not do himself; and the fact that the experimenter’s attention is focused on the “teacher” rather than the “learner” clearly communicates that the “teacher”, not the “learner”, is the actual subject of the experiment. Such highly informative demand characteristics, cues which are critical to accurate perception of the laboratory situation, are absent in the real-world. Put another way, the ecology of the experiment does not match the ecology of the real world. Because the cues unique to the laboratory situation lack ecological validity with respect to the real-world situation, the experiment itself lacks ecological validity. This is not just a matter of ecological generality – of whether the conditions in an experiment constitute an adequate sample of conditions in the real world. To the extent that the experimental situation contains cues that simply would not be found in the real world, the experiment itself lacks ecological validity. 9 Ecological validity, in Orne’s sense, is sometimes conflated with other aspects of validity. Ecological validity is an aspect of external validity, defined as the ability to generalize experimental results to different populations, situations, and variables (Campbell, 1957). Will scores on a personality inventory predict how individuals will actually behave in some specific situation? Will the performance of American college students in an experiment generalize to other populations? But ecological validity in Orne’s sense is a narrower concept, having to do with whether an experiment contains cues that preclude generalization from the laboratory to real life. This kind of generalizability is not guaranteed by mundane realism, defined as the extent to which events in an experiment are likely to occur in the real world (Aronson & Carlsmith, 1968). An experiment can employ extremely lifelike stimulus materials, in an extremely lifelike setting, collecting actual behavioral data – for example, four individuals engaged in a conversation in a park while wearing wireless EEG caps, as in Shamay-Tsoory and Mendelsohn’s (2019) Figure 1h. But the experiment would still lack ecological validity if it also contained demand characteristics that are not present in the nonexperimental situation which it is intended to represent – cues that indicate that the experimental setup is not what it seems to be; what the experiment is really all about; what the experimenter’s hypothesis is; and how the subject is expected to behave. These cues, part and parcel of the experimental setting, are not present in the real world outside the laboratory: they might, for example, lead one of the subjects to suspect that the other three participants are confederates of the experimenter. An experiment lacks ecological validity, in Orne’s sense, when the cues presented by the 10 experimental situation lack ecological validity with respect to the real-world situation that the experiment is intended to model. In introducing his ideas about ecological validity, Orne (1962) cited an early paper by Brunswik (1947/1949) on experimental design. In it, Brunswik criticized the systematic designs favored by most experimental psychologists of his day (and ours), involving only a very limited and somewhat arbitrary range of independent variables -- e.g., an experiment on size constancy with a “Noah’s Ark” design consisting of two levels of size and two levels of distance). Instead, Brunswik argued for a representative design entailing a much broader, even random sampling of stimulus conditions, which he hoped would enhance the ecological generality of experimental results. Illustrating his approach, Brunswik cited an earlier experiment on size constancy obtained size and distance estimates in 180 different, arbitrarily selected, indoor and outdoor nonlaboratory situations (Brunswik, 1944). For Brunswik, ecological generality is a quantitative matter: an experiment should include enough different stimulus conditions to ensure that it adequately samples the stimulus conditions found in the world outside the laboratory. In the experiment on size constancy just described, Brunswik sought to sample a wide range of stimulus conditions to ensure that both the cues he sampled and their intercorrelations were representative of those found in the real world. In his view, an experiment whose cues possess low ecological validity could still have high ecological generality, provided that low ecological validity was also characteristic of the cues available in the natural environment. Brunswik thought this might be 11 especially true in certain cases of social perception, such as physiognomy or judgments of personality from photographs. Ecological validity, in Orne’s revisionist sense, is a qualitative matter: experiments lack ecological validity when they contain cues (demand characteristics), unique to the experimental setting, that may threaten the ecological validity of the experiment itself. This is because demand characteristics, which provide so much information about the nature of the experiment, have no counterpart in the real world that the experiment is intended to model. With respect to the laboratory, they have a high degree of ecological validity; but they lack ecological validity with respect to the real world. Orne did not limit his concern to experiments involving deception. He invented the “real-simulator” paradigm in an attempt to disentangle those phenomena that were intrinsic to hypnosis from those which were artifacts of the demand characteristics of the hypnotic situation (Orne, 1959). Using this paradigm, he was able to show that the ostensible power of hypnosis to compel antisocial and self-injurious behavior was a product of demand characteristics unique to the experimental situation, such that the behavior of subjects in the laboratory would not generalize to the real world (Orne & Evans, 1965). Beyond hypnosis, he showed that many of the ostensible effects of sensory deprivation were produced by the demand characteristics of the testing situation, rather than deprivation per se (Orne & Scheibe, 1964). Far from using ecological validity as a cudgel to undermine experimental research, he showed how researchers could use information about demand characteristics to improve their experiments, 12 rendering them more ecologically valid (Evans & Orne, 1971; Orne & Evans, 1966). In these and other studies, Orne evinced no concern with whether his stimulus materials or tasks were lifelike. He was only interested in whether his experiments contained cues, unique to the experimental situation, that would compromise his ability to generalize from the laboratory to real life. All experiments have demand characteristics: they are the price we pay for doing behavioral research with human subjects (real-world situations have demand characteristics, too; see Kihlstrom, 2002). The only question is whether, and to what extent, those cues that are unique to the experimental situation – characteristic of the experiment but not of the real world -- render it ecologically invalid as a representation of the real-world situation we are trying to understand. Language evolves, and the technical vocabulary of psychology is no exception. The term “ecological validity” has now taken on three different meanings: Brunswik’s original concept, referring to cues; Orne’s revisionist construal, referring to experiments; and the more commonplace equation with mundane realism. This last sense has no warrant in the work of either Brunswik or Orne: even experiments employing extremely lifelike situations and tasks can lack ecological validity, depending on their demand characteristics. Orne admittedly employed “ecological validity” in a somewhat different sense than Brunswik, but his use of “ecological validity” was faithful to Brunswik’s, because the ecological validity of the cues in the experimental setting determines the ecological validity of experiment itself. 13