Observation Research

I. Goals A. Kinds of Observation Design B. Data C. Analysis D. Wrap-Up II. Kinds of Observation Design The basic idea behind observation research is to gather data in a bottom-up process. You want to avoid imposing expectations or biases on the situation to allow the relationships to emerge. Ideally, there will be no intervention from the researcher. Sometimes, however, intervention is required to get the data. The designs are arranged based on the amount of intervention by the observer (from none to a fair amount). A. No intervention Naturalistic observation. To the extent that it's possible, the observer is invisible in the situation. 1. Design considerations: In addition to providing the data of interest, this can be used to: a. Suggest hypotheses for more controlled research. Once you get some idea of what's happening in a situation, you can begin to narrow your focus to precise relationships. b. Verify work from laboratory experiments. If it's possible, you can overcome some of the limitations to generalizability from experiments by confirming the results with observation. You might ask how this type of research can be ethical given that there is no informed consent process. All studies are reviewed by an IRB, but the following properties are generally required for naturalistic observation: a. The behavior has to be publicly available with no reasonable expectation for privacy. In the Try It for Yourself section of Notes 4 one of the studies you could have evaluated (Middlemist et al., 1976) had a researcher watch men using the restroom (via a periscope device) to determine exactly when urination started. This research was controversial, in part, because people have a reasonable expectation for privacy in the restroom. b. There cannot be any identifying information collected that could violate the rules for confidentiality and anonymity. Even if people's actions are public, recording those actions and presenting them in a research report that could lead to a person's identification is another level of intrusion. A famous violation of these rules was presented in a book called Tearoom Trade: Impersonal Sex in Public Places (Humphreys, 1970). Humphreys was interested in studying anonymous sexual encounters between men in public places. Humphreys' study was actually a disguised participant observation (see below). As he said "I am convinced that there is only one way to watch highly discreditable behavior and that is to pretend to be in the same boat with those engaging in it" (p. 25). Humphreys settled on the role of "watch queen" who kept a lookout while other men engaged in sexual acts so that he could move freely around the room and observe everything that happened. Not only was he observing criminal behavior, Humphreys noted earlier in the book that he had worked in a "model institution..well stocked that summer with male homosexual patients" (p. 23), so commitment to an institution was also a possibility for his participants if they were identified. In spite of the potential harm to the participants, Humphreys recorded the license plates of men visiting these tearooms and then looked up their identities with the help of the police. He inserted his participants' names into an ongoing community health survey to interview them in their homes with their wives and children. Humphreys did make an effort to keep the identities of his participants secret, but the risk to the participants was great when the activity was criminal, could result in commitment to a mental institution, and could destroy a person's home life and career. These are the types of circumstances in which a naturalistic observation would not be appropriate. 2. Advantages: a. You can investigate things ethical considerations would otherwise forbid you from studying. b. This frees you from Hawthorne effects (people behave differently when they're being observed). Observing can cause demand characteristics (people try to figure out what the researcher wants in a situation and then do their best to make it happen). Observing can also cause other kinds of reactivity (like nervousness, embarrassment, anger) that could influence the results. 3. Disadvantages: a. Lack of control limits conclusions. b. Lack of control can make data collection difficult. 4. Study example: Baker (2000; Baker_2000.pdf) reported a naturalistic observation to see if people could tell if they were being stared at. He went to crowded places and stared at people engaged in various activities for five to 15 minutes. He recorded whether or not they seemed to notice that they were being stared at. Admittedly, this is an unusual naturalistic observation example. If people can detect stares, then the observer would be influencing the results. The goal was to answer an objection made to laboratory studies showing that people can't tell when they're being stared at. People have argued that in the natural setting participants will be better at detecting someone staring. Baker found that that did not happen. B. Some intervention Participant observation. The observer joins the group under study and plays a part in the group's interactions. 1. Design considerations: This can be: a. Disguised: Nobody in the group knows an observer is present. This has a lot of ethical implications, and would be difficult to get through an IRB review. This was the design of Middlemist et al. (1976) and contributed to its controversial reputation. b. Undisguised: Everyone knows the observer is present and collecting data (a lot of anthropology research is done this way). 2. Advantages: a. You can gain access to situations that would otherwise be closed to you. For example, if you wanted to study behavior in a group that only allowed members to participate in its rites, you'd have to join to see them. b. You can better introspect about what it feels like to be in the situation because you've experienced it (this helps with perspective taking). We normally strive for objectivity in research, but that sometimes leads to treating participants as objects of study or "othering" them. There is value in developing a deeper understanding of participants' lived experience. 3. Disadvantages: a. Reactivity. People will behave differently with an observer present. Even if it's disguised, by being in the group, the observer might unconsciously direct the group's behavior. b. You lose objectivity. By thinking like a group member you might develop sympathies for them or antipathies to them that change the observations you collect. 4. Study example: A famous psychology example of a participant observation was presented in the book When Prophecy Fails: A Social and Psychological Study of a Modern Group That Predicted the Destruction of the World by Festinger, Rieken, and Schachter (1956). Festinger et al. were interested in what would happen to a group of believers when the world didn't end in a devastating flood as they expected. They had observers join the group surreptitiously (a disguised participant observation). For example, one observer told the leader of the group that she had had a dream and "this 'dream,' completely fictitious, proved the open sesame for our female observer." The observers reported the group's activities. When the world did not end, the group concluded that their efforts had staved off the destruction. Ultimately, this incident led to the concept of cognitive dissonance. C. A lot of intervention Structured observation. You set up the context in which the observation occurs, and then let it happen naturally. For example, if I wanted to assess the attachment style of a child, I could use the "strange situation" (Ainsworth, 1978). A series of increasingly more challenging situations is used with a particular response expected to each based on attachment style. The idea is that the particular event observed is structured, but the behavior of the participants is “natural.” 1. Advantages: a. You can cause infrequent events. It might take a long time to see how a parent interacts with their child if you just wait for it, but this allows you to make it happen. b. You can test the limits of a person's abilities. For example, if Piaget found that a child could conserve volume, he would move on to a new task to find out just what each child could do. 2. Disadvantages: a. Not as natural. 3. Study example: Solomon et al. (2019; Solomon,_Beetz,_Schoberl,_Gee,_&_Kotrschal_2019.pdf) extended the strange situation to companion animals (service dogs) to assess their attachment style. You can see the set-up of their observation in Figure 7.1. Figure 7.1. The set-up of the testing room for Solomon et al. (2019). To test the dogs' reactions, the first episode was for the caregiver and dog to be in the room alone. Then a stranger came in. Other conditions were for the caregiver to leave the dog alone with the stranger, to leave the dog alone in the room, and to have the stranger come in while the dog was alone. Behavior in each episode was recorded to assess the dogs' attachment styles. For example, secure attachment was when the dog approached the caregiver when they came back in the room and sought contact. As you can see, the situation is structured to allow all of the different episode types to be condensed into a single session. However, the behavior is being observed rather than being manipulated. You can practice identifying the various types of design in comprehension check 7.1. Comprehension check 7.1 D. Terms 1. Naturalistic observation 2. Hawthorne effects 3. Demand characteristics 4. Participant observation 5. Reactivity 6. Structured observation Comprehension check 7.1 evaluation: You can check your answers here. Note that there are multiple ways to design studies that can all be correct. a. Do adults still show problems with conservation of mass, number, etc. if you frame the question correctly? This feels like a structured observation. I guess you could technically do it as a naturalistic, but it would take forever to see the same person do multiple types of task with different levels of challenge. b. Are people more likely to make a legal stop at a stop sign if other people are in the area? This feels like naturalistic. You could do it as structured and set up various interactions, but it doesn't seem necessary. c. What territorial markers do people use to establish "their" space in public? Naturalistic; you could get the data from watching people. d. How do people respond to a territorial intrusion on "their" space in public? Naturalistic could work, but you may not get many intrusions. Participant observation would probably accelerate data collection. e. Are people more likely to pick up trash if they see someone else pick up trash? Structured makes the most sense in that you could set up various situations to see how people react. f. How do people respond to various types of messages from a group leader about outsiders? This feels like participant since it's likely the group's activity would be private. III. Data A. What to record Make a narrative record, which is a faithful reproduction of everything that happened (video or audio tape if you can get it). Why record it all? Because you usually don't know what's important. If you're selective, you might overlook the most important thing. You can always condense the record once you have it. It helps if you follow a research protocol. This is a specific description of how the measurements are to be made. This helps to eliminate random or systematic errors in recording the data. Figure 7.2 contains a sample record from the Tearoom Trade (Humphreys, 1970) study. Figure 7.2. Sample observation record sheet from Humphreys (1970, p. 35). What to record? 1. Setting: What is the environment around the observation? Include anything that could influence the participants' behavior. 2. Participants: Who's there? Record all characteristics of these that might be relevant. 3. Events: What happens? Events can be very difficult to define. It is a really good idea when setting up an observation to evaluate the literature and make sure you are measuring and defining things appropriately. In a sense, we can think of these three as independent variables. They are not manipulated, but they serve as the inputs to behavior. 4. Behaviors: These are your main data. What do the people being observed do in response to the events that take place? Two points about these records: a. Record them immediately. You should avoid relying on memory if at all possible. b. Avoid interpretation. We don't want personal biases to get in the way of the data. For example, if you see one person hit another person, you don't want to write down anything about their emotional states, just the fact that someone got hit. Try It for Yourself Parking lots can be very territorial places, as shown in this clip from Fried Green Tomatoes. Video clip 7.1. Parking lot scene from Fried Green Tomatoes. Ruback and Juieng (1997; Ruback_&_Juieng_1997.pdf) were interested in a different sort of parking lot territoriality. Do people punish waiting drivers when backing out of parking spaces? Many people believe that others take longer to back out when they are waiting, Ruback and Juieng wanted to confirm those impressions. For their Study 1, Ruback and Juieng (1997) did a naturalistic observation. They went to the parking lot and observed how long people took to back out, both with and without waiting drivers. They found that people do take longer when someone is waiting (however, causal conclusions are not possible as to why people take longer). Your assignment is to design your own version of Ruback and Juieng's (1997) Study 1. 1. What characteristics of the setting will you record? 2. What characteristics of the participants will you record? 3. The event is backing out of a parking space. How will you define that event (when does it begin and when does it end)? When you get your design together, compare your work to Ruback and Juieng (1997). Focus on the things you thought of that they didn't, or things they thought of that your didn't. When you're finished, put your design and your comparison in the discussion forum Notes 7 Try It for Yourself comprehension check. B. Terms 1. Narrative record IV. Analysis A. Descriptive statistics The first step in any analysis is to describe the data. I'm going to present a rudimentary summary here, just to remind us of the highlights. To get some data, I asked my class how many previous psychology courses they had taken. Those data are presented in Table 7.1. Participant Prior Number of Classes 1 4 2 3 3 6 4 3 5 4 6 5 7 13 8 4 9 4 10 5 11 5 12 4 13 5 14 6 15 5 Table 7.1. Sample data to use to compute descriptive statistics. 1. The first step is to visualize the data and make sure there are no issues. For this, we will make a frequency distribution. We plot each score on the xaxis (how many classes people have taken) and the frequency with which that score occurs on the y-axis. The frequency distribution for our data is presented in Figure 7.3. Figure 7.3. Frequency distribution for the data in Table 7.1. If we didn't have that person with 13 classes, the data would look pretty good. We want the frequency distribution to approximate a normal distribution (symmetrical with a hump in the middle and low at both ends). Because of the 13, we will have a positively skewed distribution (the best fitting curve will tail off towards the positive numbers). The 13 is an outlier (it's outside of the bulk of the data; what to do with outliers is a problem researchers frequently face). Having an outlier affects the choices we will make for the rest of the descriptive statistics and could influence our choice of test (e.g., a t test is relatively robust for this kind of problem, but others might not be OK). 2. The second step is to find the middle of the distribution. There are three measures for this (mean, median mode). If you have a normal distribution, these are all the same number. a. Mean: This is the most useful. It is the average. Add the scores, divide by the number. For us, the mean is 5.07. (Without the 13, the mean is 4.5; you can see how the outlier pulls the mean towards itself.) The mean is the descriptive statistic that is usually reported for the center. b. Median: This is the score in the middle. For us, the median is 5. The median is less affected by outliers since it's only looking for the score in the middle. c. Mode: The mode is the most frequent score. We have two, 4 and 5. 3. The third step is to describe how spread out the scores are. We have three statistics for this (two are related): a. Range: What values do the scores cover? Ours go from 3-13, or 11 scores are in the range. Range really only takes the ends into account, and not the rest of the scores. b. Variance: The variance takes all of the scores into account. It is the average squared distance from the mean. To calculate it, subtract the mean from every score, square those differences, add them up, divide by the number of scores (to estimate the population variance, divide by the number of scores 1). Our sample variance is 5.26. The variance is a very useful number. It is our estimate of all of the differences in the dependent variable. This is what we're usually trying to explain. For example, on the first exam, not everyone got the same score. Why not? The variance represents the total differences, and our statistical procedures will allow us to try to explain those differences and answer the question. c. Standard deviation: This is the square root of the variance (2.29). This puts the variance back in the original units. The standard deviation is the descriptive statistic that is usually reported for the spread. B. Chi-square After an observation study, you can have interval or ratio data for some of your measures (e.g., in the Tearoom Trade study they recorded how long men waited in their cars before coming in; time is ratio). A lot of the time, however, your observation data will be frequencies because nominal variables are easier to observe. You can compare the frequency distributions of groups of people observed using chi-square. Let's say that I'm interested in how people's territorial markers change depending on where in the room they are. Markers are how people "claim" a public territory and signal to others not to intrude. Will these differ based on the value of the territory being claimed (edge of the room vs. middle)? To get the data, I did a naturalistic observation in a crowded space in the library. When a person came in I noted where in the room they sat and their first choice of territorial marker. Table 7.2 contains some sample data. fo Books Spread Out Book Bag on Chair Move Furniture Total Around the Edge 10 2 13 25 In the Middle 10 11 4 25 Total 20 13 17 50 Table 7.2. Observed frequency distributions of territorial markers as a function of location in the room. The top row is the frequency distribution of territorial markers for people on the edge of the room and the bottom row is the frequency distribution for people in the middle of the room. If you look at those distributions, you can see that they seem to differ. Chi-square will allow us to assess those differences. The formula for chi-square is presented in Equation 7.1. Χ2 = ∑ [(fo - fe)2 / fe] Equation 7.1. The formula for a chi-square statistic. In the formula, fo is the observed frequency. That is the number we observed when we collected the data. The fe is the expected frequency. This is the frequency we would have seen if the choice of marking strategy were independent of the position in the room (there is no relationship between position and marking strategy). Where does expected frequency come from? The formula for expected frequency is presented in Equation 7.2. fe = row total * column total / grand total Equation 7.2. The formula for each cell of the expected frequency distribution. For example, for the first cell (the intersection of "books spread out" and "around the edge"), the numbers are 25 (row) * 20 (column) / 50 (grand total) = 10. You can see the expected frequencies in Table 7.3. fe Books Spread Out Book Bag on Chair Move Furniture Total Around the Edge 10 6.5 8.5 25 In the Middle 10 6.5 8.5 25 Total 20 13 17 50 Table 7.3. Expected frequency distributions of territorial markers as a function of location in the room if the two variables were independent. How is chi-square computed? We can transpose the scores into a table like Table 7.4 to help with the math. In the first column I put the observed frequencies from Table 7.2. In the second column I put the expected frequencies from Table 7.3. The rest of the columns are just steps based on the formula. Adding up the values in the final column gives me the value for the chi-square. (Note that the fo - fe column should add up to zero; this is a useful check on your math to that point in the analysis.) fo fe fo - fe (fo - fe)2 (fo - fe)2 / fe 10 10 0 0 0 2 6.5 -4.5 20.25 3.11 13 8.5 4.5 20.25 2.38 10 10 0 0 0 11 6.5 4.5 20.25 3.11 4 8.5 -4.5 20.25 2.38 ∑=0 Χ2 = 10.98 Table 7.4. Computation of the chi-square for the territorial marker by position in the room observation. The more the two frequency distributions differ, the bigger the chi-square will be, and the more likely it is that the two distributions are not independent. To tell if the chi-square is significant, I need to look up chi-square in a table with the right degrees of freedom. The formula for degrees of freedom is presented in Equation 7.3. df = (number of rows - 1) * (number of columns - 1) Equation 7.3. The formula for the degrees of freedom of a chi-square analysis. For us the degrees of freedom are (2 - 1) * (3 - 1) = 1 * 2 = 2. To see if the value is significant (different from what we would expect by chance), I Google "critical values chi-square" and get a table of critical values. Looking in the table, I find the critical value for alpha = .05 and 2 degrees of freedom to be 5.99. My value is higher, so it's significant. The final step is to interpret that result. As I noted before, this class can help with a local interpretation. What can we say after a chi-square analysis? I've put the interpretations in Table 7.5. As for what it means, that depends on the research question that led you to look at territorial markers in different parts of the room in the first place, and that is beyond the scope of this class. Outcome Conclusion Loosely Speaking Chi-square significant "There is enough evidence to conclude that the two frequency distributions are not independent." There is a relationship between the two variables Chi-square not significant "There is not enough evidence to conclude that the two frequency distributions are not independent." There is not a relationship between the two variables (not enough evidence for a relationship). Table 7.5. What to conclude depending on the outcome of your chi-square analysis. How would you write this up? A sample paragraph for our study is: "The distributions of territorial marking strategies for people around the edge of the room and people in the middle of the room were compared using a chisquare test of independence. The frequencies are presented in Table 7.2. The chi-square was significant, Χ2(2, N = 50) = 10.98, p < .05. There was enough evidence to conclude that the two frequency distributions were not independent. Participants around the edge of the room were more likely to move furniture and participants in the middle of the room were more likely to put a book bag in a chair." C. Terms 1. Frequency distribution 2. Outlier 3. Mean 4. Median 5. Mode 6. Range 7. Variance 8. Standard deviation 9. Chi-square V. Wrap-Up A. Progress We have made some progress on the data collection portion of the class in that we've finally collected some data. B. Five big things We've added two things to the five big things I want you to take away from the class. Here is the updated list: 1. Correlation does not imply causation. Do not be tricked. 2. Go to the library. This theme is going to continue to emerge, but it should already be obvious that you make a lot of decisions in a research project. What to measure? What to record? How to record it? How to analyze it? For each, guidance from the literature will make you more likely to be successful. On the flip side, when you have empirical questions, the literature will be the place to look for answers. 3. There are no right answers, only reasons (in real life there is no answer key). One of the hardest transitions to becoming a professional is that you don't have someone "grading" your work and telling you if you got the right answer. You have to apply your best thinking (and a review of the relevant literature) and hope for the best. This showed up here in trying to identify various types of designs. Reasonable people can come up with different answers, and more than one answer can be correct.