Surveys and Observations - Describing What People Do

Chapter 6: surveys and observations: describing what people do  Construct validity of surveys and polls: o Choosing question formats o Writing well worded questions o Encouraging accurate responses  Clicker: a survey question that has you rate a response from agree to disagree is an example of: likert scale  Clicker: a response set can be in the form of o Yay saying answers  Choosing question formats o Open ended questions o Forced choice format o Likert scale o Semantic differential format ( example shown)   Confusing - crystal clear (numbers inbetween) Writing well worded questions o How can you write items that avoid these problems?   Leading questions  Double barreled questions  Negative wording Encouraging accurate responses o What are some solutions to these problems?  Giving meaningful responses efÏciently  Using short cuts   Yea saying  Fence sitÝng Trying to look good   Saying more than we know  Reporting memories of events Clicker: Masked or blind studies are designed to deal with o Observer bias  Construct validity of behavioral observations o Examples of frequency claims based on observational data   Inclusion of filler questions (this is self-report) “How many words do you say in a day” need to do this observationally. o Are observations better than self-reports  Making reliable and valid observations:  Observer bias: when observers see what they expect to see. Potential threat to construct validity. Observers may see what they expect to see, even if observers are all watching the same behavior. o All watch same video, but set it up so their would be biased.  Observer effects (expectancy effects): when participants confirm observer expectations. o Knew all of scores on exam 1, everyone who got an A or B sit there, C in middle, then TA make observations. o Observers being biased effects participants are preforming because you know they know. o Observer bias on steroids.  Reactivity: when participants react to being watched o Not biased exactly. o Found out president is coming in to observe class, it would change out behavior.  What would you do to make observations reliable and valid  Try blending in  Waiting it out so it effects behavior less  Results of behavior and outcome o Good observations are both reliable and valid   Having a clear coding manual can help interrater reliability  Behavioral observations need good interrater or intercoder reliability  How much did two raters agree with one another Masked/blind studies o Clicker: Masked or blind study is designed to handle:  Observer bias/ observer effect   Biased because they know whose been in jail or etc, responding to that bias its effecting your behavior it would be better if study was masked where observer doesn’t know this information. Not knowing who is in which group. o Clicker: D. ewill, a developmental psychologist ….   A, Reactivity Self-report or observation? o A political psychologist wishes to measure the percentage of people who vote in elections in a rural area, self-report or observation? How you would measure the variable?  Use observational measure, operationalize variable, multiple coders see how many people, Observational, get number of people who voted in that variable from polling stations, o Clicker quiz 6:  This chapter is about   Construct validity Suppose you are creating a survey for research methods to find out what students think about collaborative learning. Which of the following will be important in terms of the construct validity of your survey ?  All the above ( choosing appropriate question formats, writing well worded questions, encouraging accurate responses)  Dr. sorbellini is conducting an observational study of college students cell use. Which is NOT something to take into consideration when making reliable and valid observations?   Response set Which is a true statement?  Observations CAN be better than self-report Surveys and observations: describing what people do  Surveys, polls, and observational methods are used to support frequency claims, but they also measure variables for association and causal claims.  Construct validity of surveys and polls  (these two words can be interchangeable. Survey and poll both mean the same thing; a method of posing questions to people on the phone, in personal interviews, on written questionnaires, or alone. o Survey:  is often used when people are asked about a consumer product  Question formats include open ended, forced choice, Likert scale and semantic differential. o Poll: is used when people are asked about their social or political opinions.  Choosing question formats ( four basic formats) o Open ended questions:  Researchers may as open ended questions that allow respondents to answer any way they like.  The drawback is that the responses must be coded and categorized, a process that is often difÏcult and time consuming.  For efÏciency, researchers in psychology often restrict the answers people can give o Forced choice questions  One specific way to ask survey questions  People give their opinion by picking the best of two or more options  Used in political polls, o Likert scale  Contains more than one item and each response value is labeled with the specific terms “strongly agree, disagree, agree” is called likert scale. o Semantic differential format  Instead of degree of agreement, respondents might be asked to rate a target object using a numeric scale that is anchored with adjectives; this is called a semantic differential format   Ex: ratemyprofessor Question wording matters o Leading question  One whose wording leads people to a particular response because it explains why some people oppose the law. o Double barreled questions  A wording of questions is sometimes so complicated that respondents have trouble answering in a way that accurately reflects their opinions.  It asks two questions in one.  Poor construct validity because people might be responding to the first half of the question, second, or both. Careful researchers would have asked each question separately.  “do you agree that the second amendment guarantees your individual rights to own a gun  DifÏcult to answer in a valid way o Negatively worded questions  Another way survey items can be unnecessarily complicated. Whenever a question contains negative phrasing, it can cause confusion therby reducing the construct validity of a survey or a poll.  When possible, negative wording should be avoided, but researchers sometimes ask questions both ways  After asking the question both ways, the researchers can study the items internal consistency (using cronbach’s alpha) to see whether people respond similarly to both questions.  Can reduce construct validity because they might capture people ability or motivation to figure out the question rather than their true opinions.  DifÏcult to answer in a valid way o Question order  The order in which questions are asked can also affect the responces to a survey.  The most direct way to control for the effect of question order is to prepare different versions of a survey, with the questions in different secquences.  Response sets o Short cut respondents take when answering survey questions. Rather than thinking carefully, people might answer all of them positevly, negatively, or neutrally.  Weaken construct validity because these survey respondents are not saying what they really think o Our common response set is ACQUIESCENE  When people say yes or strongly agree to every item without thinking. o Specific response set: Fence sitÝng  Playing it safe by answering in the middle of the scale, especially when survey items are controversial.  People might also answer in middle when a question is confusing or unclear.  Weaken a surveys construct validity when middle of the road scores suggest that some responders don’t have an opinion, when they actually do.  Approach is to take the neutral option  Common way to get people off the fence is to use forced choice questions. One of two answers  Trying to look good  Socially desirable responding/ faking good  Survey respondents give answers that make them look better then they really are, these responses decrease the surveys construct validity. o To avoid, a researcher might ensure that the participants know their responders are anonymous.  Faking bad  Will not tell the truth on a survey or other self-report measure, a similar but less common phenomenon is called faking bad. Chapter 6 book notes  What type of validity is most important to consider when creating a poll or survey? o Construct validity  Forced choice questions ask people to choose between two or more options o Respondents are asked to choose a number corresponding to how much they agree in Likert scale o Responders can happen any way they like in open ended questions  When observer’s knowledge of the research hypothesis influences how they record their observations observer bias occurs. When participants change their behavior based on perceived expectations from the observer, its called observer effects. When people change their behavior based ni the presence of someone else, this is known as reactivity. o Preventing observer bias and observer effects  Researchers must ensure the construct validity of observational measures by taking steps to avoid observer bias and observer effects.  Codebooks; clear rating instructions. So observers can make reliable judgements with less bias.  If respondents become fatigued, or are unengaged in the survey, they may answer questions in a general pattern without really considering their honest responses. This is known as using a response set. If respondents continually select “yes”, they are showing acquiescence. If respondents select all middle options, that would be referred to as fence sitÝng. With sensitive topics, sometimes respondents will answer in such a way that their thoughts are in line with what they believe everyone else’s are. This is called socially desirable responding.  Reactivity example: o Small children stop and stare at the experimenter taking notes in the room  Observer bias example: o Researchers interpret prisoners behavior as more aggressive than college students behavior even though the behavior is identical.  Observer effects: o A participant is unusually polite because he thinks the study is about social interactions.  Fence sitÝng: o Playing it safe and picking the middle of the scale  Socially desirable responding: o Most of us want to look good to others, but when survey respondents give answers that make them look better than they really are, these responses decrease the surveys construct validity. They will not tell the truth on a survey.  Acquiescence o “yes” or strongly agree to every item instead of thinking carefully  Samantha is conducting an online survey that asks participants many different questions about personality and takes nearly an hour to complete. After collecting some pilot data, Samantha notices toward the end of the survey that participants are choosing “strongly agree” for every question. o Add reverse worded items to improve the survey Chapter 7:Sampling: estimating the frequency of behaviors and beliefs (external validity)  Generalizability: does the sample represent the population o Populations and samples  Population is a larger group from which a sample is drawn  Sample is intended to represent that population o Obtaining a representative sample: probability sampling techniques o Settling for an unrepresentative sample: biased sampling techniques  Probability sampling techniques o Simple random sampling o Cluster sampling and multistage sampling o Stratified random sampling  Researchers identify a particular demographic o Oversampling o Systematic sampling  Combining techniques  Biased sampling techniques o Convivences sampling o Purposeful sampling o Snow ball sampling o Self selected samplings Clicker: dr oishi is an educational psychologist interested in students attitudes toward math and the effect of those attitudes on performance on standardized tests…..  This is an example of Which of the following sampling techniques o Multistage