Ch. 3 Three Claims, Four Validities Interrogation Tools for Consumers of Research

Ch. 3 Three Claims, Four Validities: Interrogation Tools for Consumers of Research      What IS a variable? What is the difference between a measured and a manipulated variable? Review: what is the difference between an independent variable and a dependent variable? What is the difference between a conceptual and an operational definition? Be able to distinguish between the three claims research can make and recognize examples. o What are the four types of association claims? o WHY doesn’t correlation equal causation? AKA: what criteria need to be present to establish causation? Be able to differentiate between the four types of validities discussed and relevant issues o Ex: how does random assignment affect external validity Claims in research:  How do you know if claims made in a research article or in the popular press are accurate?  What are the different types of claims?  When can you make them? Three Claims: 1. Frequency Claim o 1 = Never, 2 = A little, 3 = Sometimes, 4 = A lot, 5 = Always 2. Association Claim o Correlations o Correlation study 3. Casual Claims o Factor A  Factor B Variables Variable: (constructs)  Is something that varies, so it must have at least two levels, or “values”  Example: o “More Than 2 Million U.S. Youths Depressed”  Variable  Depression  2 levels  Depressed and not depressed o “ADHD Drugs Not Linked to Future Drug Abuse”  2 variables  ADHD meds, drug abuse  2 levels of each variable  (1) Medicated for ADHD and (2) Not medicated for ADHD  (1) Abuses drugs and (2) Does not abuse drugs Constant:  Is something that could potentially vary but has only one level in the study in question o Example: In research on fathers, sex is not a variable Measured and Manipulated Variables  Measured Variable: o Dependent variable o Is one whose levels are simply observed and recorded o In each case, measuring a variable is matter of recording an observation, a statement, or a value as it occurs naturally o Examples:  IQ, sex, height, socioeconomic status  Stress, depression (would need way to measure this)  Manipulated Variable: o Independent variable o A variable a research controls, usually by randomly assigning participants to the different levels of that variable  Sometimes called treatment levels o Examples:  Drug dosage, environmental stimuli, etc.  Some variables cannot be manipulated – they can only be measured o Example: gender, IQ, etc From Conceptual Variable to Operational Definition When researchers are discussing their theories and when journalists write about research—they use concept-level language  Conceptual Variables: o Abstract concepts  Such as: “Shyness” or “intelligence” o Sometimes called a “construct”  Conceptual Definitions: • Researcher’s definition of a variable at an abstract level/ theoretical level at which conceptual variables must be defined at To test hypotheses, researchers have to do something specific in order to gather data  Operational variables: o Also known as “operationalization”  The specific way in which a concept of interest is measured or manipulated as a variable in a study o Operational Definitions:  When testing their hypotheses with empirical research, they create operational definitions of variables o Importance:  Variables should be defined in terms of the operations needed to produce them.  Allow others to replicate your research  Example: - “Subjects received a reward for each appropriate response.” VS. “Rats received a 45 mg Noyes Formula A food pellet for every bar press in the operant conditioning chamber.” o Operationalizing definitions/ To operationalize:  Means to turn a concept of interest into a measured or manipulated variable  Some variables start out as abstract concepts  Depression     Stress  Reward Need measureable (Operational) definitions  Can use specific numerical criterion  Can use structured set of questions with associated scores Some are easy  Weight gain or weight loss  External reward  ADHD meds Some are not  Depression  Intellectual abilities  Drug abuse o Examples:  Variable: Aggression  OD #1 (Physical): Self-reported of number of physical/violent altercations in the past six months  OD #2 (Social): Have you ever purposely tried to make a coworker feel bad at your job?  Variable: Chess Expertise  OD: The top 1% of all United States Chess Federation (USCF) tournament chess players.  Variable: Reasoning Ability  OD: Score on the Ravens Progressive Matrices (IQ test) Three Claims Claim  It the argument someone is trying to make Frequency Claims  Def. o Describe a rate or level of something or degree of a single variable  Generally focus on only 1 variable o Association and causal claims focus on relations between more than 1 variable  The variables are always measured, not manipulated  Not to be confused with anecdotal claims o Tell a story (true, perhaps), but don’t report results from a study  Potential problem: o What is “a lot” to one person may be “a little” to someone else… Association Claims  Def. o Argues that one level of a variable is likely to be associated with a particular level of another variable  Variables that are associated are sometimes said to correlate o Meaning that when one variable changes, the other variable tends to change, too  Variables are measured not manipulated  Examples: o Belly Fat Linked to Dementia o ADHD Drugs Not Linked to Future Drug Abuse  How to represent: o Scatterplot:  A graph in which one variable is plotted on the y-axis and the other variable is plotted on the x-axis  Each dot represents one participant in the study, measured on the two variables  Types: 1. Positive  High goes with high  Low goes with low 2. Negative  High goes with low  Low goes with high 3. Zero  No association 4. Curvilinear  In a curvilinear correlation, one variable changes its pattern as the other variable increases.  As self esteem increases, GPA increases, and then decreases.  As age increases, healthcare visits decrease, then increase.  A curve  Making Predictions Based on Associations / Why are they useful? o Correlation does not imply causation  Third-variable problem  There may be an unmeasured variable that actually causes variables to covary (change together)  Directionality problem  Not always possible to specify the direction in which a causal arrow points  o Example: Do video games and the internet ruin our relationships? (Padilla-Walker et al., 2009) Don’t know what causes what o Correlation does allow prediction  If you know there is a positive or a negative correlation, then given the value of one variable, you can predict the value of the other variable  Abdominal fat and dementia  Cell phone and sperm quality  The stronger the relationship between the two variables, the more accurate the prediction should be Casual Claims  Causal claims argue that one variable is responsible for a change in the other  Can be positive, negative, or curvilinear  Example: o Music Lessons Enhance IQ o Debt Stress Causing Health Problems, Poll Finds o Family Meals Curb Teen Eating Disorders  Causal claims are stronger, and therefore held to a higher standard o Must establish that two variables are correlated o Must show that one came first (causal variable) and other came second (outcome variable) o Must establish that no other explanations exist for the relationship  Experiments are required to determine this; must manipulate one variable and record the other  Manipulate one variable (Independent Variable); Measure other variable (Dependent Variable)  Important considerations for such experiments  Random selection of participants  Random assignment to groups  What sorts of 3rd variables might need to be ruled out in this study?  How does random assignment address the 3rd variable problem? Interrogating the Three Claims Using the Four Big Validities Validity  Refers to the appropriateness of a conclusion or decision  In general a valid claim is reasonable, accurate, and justifiable o Is a claim valid?  Is a claim accurate? Reasonable? Justifiable?  Types: o Construct Validity  Refers to how well a conceptual variable operationalized  Measures how well the variables in a study are measured and manipulated  Question  How well did a study measure a variable? - “Was it operationalized appropriately?” o Statistical Validity  Use statistical validity to describe their data  Also called “statistical conclusion validity”  Def.  Is the extent to which a study’s statistical conclusions are accurate and reasonable  Statistical validity varies depending on claim  Question:  Are the results statistically accurate? o Internal Validity  “Third-variable criterion”  Only relevant for causal claim  Question:  How likely is it that one of the variables (as opposed to something else) caused the change in the other?  In a relationship between one variable (A) and another variable (B), the extent to which A, rather than some other variable (C), is responsible for the effect on B  Internal Validity:  Changes in the dependent variable are attributed to the manipulation of the independent variable - We want to insure that the experiment has internal validity o External Validity  Refers to how well a study is generalized  Measures the extent of which the results of a study generalize to a larger population as well as another situation  Question:  How well do the results generalize, or to represent, people besides those in the study itself? Interrogating Frequency Claims  Construct Validity of Frequency Claims: o Question to consider:  How well the researchers measured the variables? o Number of ways to operationalize a variable – some better than others o Construct Validity:  When you ask how well a study measured or manipulated a variable o To ensure construct validity:  Research must establish that each variable has been measured reliably  Be sure the measure gives similar scores on repeated testing  That different levels of a variable accurately correspond to true difference  External Validity of Frequency Claims: o Generalizability:  How did the researcher choose the study’s participants, and how well do those participants represent the intended population?  Examples of Construct validity and external validity: o Construct:  “Two million U.S. Teens Depressed.”  The variable needs to be measured reliably (similar scores on repeated testing) o External:  “Half of Americans struggle to stay happy”  How did the researchers choose their participants?  Do those participants accurately represent the population?  Statistical Validity of Frequency claims: o Percentages usually accompanied by margin or error estimate  Def. A statistical figure, based on sample size for a poll, that indicates where the true value in the population lies o Asking statistical validity of frequency claims – means reminding yourself that the number associated with the claim is an estimate, and it has a special amount of error with it  Internal Validity of Frequency claims: o Usually not asserting causality so not internal validity relevant Interrogating Association Claims  Construct validity: o Construct validity questions asses how well such measurements were conducted o How well the researched measured each variable  External validity: o To what population, settings, and times can we generalize this association claim? o How representative is the sample? o To what other problems might the association be generalized?  Examples of Construct validity and external validity: o Construct:  Heavy Cell Phone Use Tied to Poor Sperm Quality  How did they measure phone use?  How did they measure sperm quality? o External:  Do the data generalize to other populations, times, cultures, contexts, etc.?  Statistical validity: o Extent to which conclusions are accurate and reasonable o Question:  How strong is the association? o Types of error:  Type I error:  When a study concludes that there is an association between two variables when there is really no association  “False positive”  Type II Error:  When a study concludes that there is no association between two variables when there is really an association in the full population  “Miss” o Internal Validity:  Association claims do not assert casualty  so not relevant Interrogating Association Claims  Three Criteria for Causation: (3 techniques) 1. Covariance (also true for association claims)  Refers to the two variables being related  But must do more than just show that the two variables are associated 2. Internal validity  “Third –variable criterion”  A study should be able to eliminate alternative explanations for the association 3. Temporal Precedence  One variable comes/changes before the other variable  Example:  “Music lessons enhance IQ” - If this was an association, could it go in the other direction?  “Debt stress causes health problems” - If this was an association, could it go in the other direction?  Experiments can support casual claims: o Experiment:  One variable is manipulated and the other is measured o Independent variable:  Manipulated variable o Dependent variable  Measured variable o Example:  To see whether music effects IQ  IV: - Music lesions  DV: - IQ o Random Assignment:  Increases internal validity by allowing to control for potential alternative explanations Prioritizing Validities  Hard to design experiments that: o Measure perfectly  Construct validity o Generalize broadly  External validity o Control all extraneous variables  Internal validity o Leave no room for error  Statistical validity  So what’s a researcher to do? 1. Prioritize based on goals of study  Internal > external?  External validity not always possible to achieve  Example: - When focusing on internal validity and using random sample research may not try to sample a large population but just a random sample of a school  External > construct?  Example: - Researcher wants to conduct an experience on a large population -- did not have money or time to conduct a large question measure of a variable so have to use the smaller measure - Sacrifice construct for external 2. What are you focusing on in YOUR OWN projects as you are developing your scale items?