Three Meanings of “Ecological Validity”

Three Meanings of “Ecological Validity” Hammond (1998) criticized social psychologists and others for misusing the concept of “ecological validity”. He correctly stated that in, coining that term, Brunswik was referring to the validity of the cues available for perception and judgment, not the validity of experiments. The issue came up again recently in the pages of Perspectives on Psychological Science (PPS), when Simone Shamay-Tsoory and Avi Mendelsohn (2019) published an article advocating for the use of more “ecologically valid” research in cognitive neuroscience. In reply, Gijs Holleman and his colleagues revived Hammond’s critique (Holleman, Kemner, & Hessels, 2020). The alleged misuse of the concept of ecological validity had its origins in a classic paper by Martin Orne (1962) on the social psychology of research in experimental psychology. As a former graduate student of Orne’s, I had long contemplated writing a response to Hammond’s critique, so I decided to join the current thread with a note attempting to explicate what Orne had in mind (Kihlstrom, 2020). In it, I agreed that the common usage of “ecological validity”, referring to something like mundane realism, is indeed inconsistent with Brunswik’s intentions (and Orne’s for that matter). But I also argued that Orne’s revisionist usage is consistent with Brunswik’s, because Orne emphasized the information provided to subjects about the true nature of 1 an experiment – cues that Orne called ”demand characteristics” I do not know whether Orne ever met Brunswik. Orne, then based at Harvard Medical School, did teach briefly at Berkeley, but that was during the summer of 1962, after Brunswik’s death (1955). On the other hand, Orne’s family emigrated from Austria to New York at the time of the Anschluss, when he was about 11 years old. Orne’s mother, Martha Brunner-Orne, a psychiatrist, may have known Brunswik and his wife in Vienna (Else Frenkel-Brunswik, a psychoanalyst who worked with Leon Festinger on The Authoritarian Personality, died in 1958). Orne never offered his own definition of ecological validity, and his only reference citation to Brunswik (Orne, 1962, fn. 4) was to a long article that the latter had prepared for his course on research methods (it is difficult to imagine most of today’s graduate students, much less undergraduates, getting through it). That document is still available in the UC Berkeley Library, and as far as I can determine it is identical to Brunswik’s published contribution to a symposium on probability and statistics (Brunswik, 1947/1949). In his footnote, Orne characterizes ecological validity “in the sense that Brunswik (1947) has used the term: appropriate generalization from the laboratory to nonexperimental situations”. Such a phrase does not actually appear in the 1947 article, and the closest that Brunswik comes to it is in a discussion of ecological (or situational) generality, achieved through the representative design of psychological 2 experiments in such a way as to insure that cues available in the real world are adequately sampled in the experimental setting. For Brunswik, however, ecological generality is just a matter of insuring that, in an experiment on size constancy for example, the sizes and distances involved are an adequate sample of those that would be encountered in the real world outside the laboratory. Orne’s revisionist concept of ecological validity goes beyond representative design. Instead, he argued that experiments sometimes contain cues – demand characteristics – that simply aren’t present at all in the real-world setting. These cues are ecologically valid in the experimental setting, in that they provide information to a subject about the true nature of the experiment. But because they are unique to the experimental situation, they are not ecologically valid with respect to the real-world setting that actually interests the researcher. To the extent that subjects utilize demand characteristics in the experimental situation which have no counterpart in the real world, their behavior in the experiment will not generalize to the real world. In part, Hammond’s (1998) critique was justified: the familiar equation of “ecological validity” with mundane realism is indeed inconsistent with Brunswik’s coinage. But it is also not what Orne (1962) had in mind. For Orne, experiments do not lack ecological validity when they fail to use lifelike stimulus materials and tasks, or even when they fail to predict behavior in the real world. Experiments lack ecological validity when they provide cues to the subject that the experimental situation is not what it appears to be, or as presented by the experimenter. Because the ecological validity of an experiment depends on the ecological validity of the cues it provides to the subject, Orne’s revisionist construal is broadly consistent with Brunswik’s intentions. 3