Ch. 5 Identifying Good Measurements

Ch. 5 Identifying Good Measurements  Review: make sure you understand the challenges associated with operationalizing variables o What are the advantages/disadvantages of using self-report, observational, physiological measures to operationalize a variable? Understand the 4 scales of measurement: nominal, ordinal, interval, ratio o Be able to identify when each type of scale is being used. o What are the advantages/limitations of each?    o How do you identify whether a continuous variable uses an interval or ratio scale? Know the three types of reliability and when they are most relevant? (E.g. when should you look for internal reliability vs. when should you look for inter-rater reliability?) o How can a scatterplot be used to evaluate reliability? o Are correlations related to reliability? Understand the various types of validity o What is the difference between content and construct validity? o What is the usefulness of “Known-Groups Evidence” for Predictive & Concurrent Validity? o How can someone determine whether there is convergent or discriminant validity between two variables? Ways to Measure Variables Three Common Ways to Measure: 1. Self-report 2. Observational 3. Physiological  Construct / Conceptual Variable o Researchers definition of the variable in question at a theoretical level o Construct validity:  A measurement of how well a variable was measured (or manipulated… but that’s later) in a study  Operational Variable: o Measured variables:  Variables whose levels are observed and recorded (with no manipulation)  Conceptual and operational definitions: are you “happy”? o “Happiness”: subjective well-being  Conceptual definition o How do we operationalize this?  Maybe responding to multiple items using a 7-point scale where 1=strongly disagree and 7=strongly agree…  ___ In most ways my life is close to my ideal.  ___ The conditions of my life are excellent.  ___ I am satisfied with my life.  ___ So far I have gotten the important things I want in life.  ___ If I could live my life over I would change almost nothing.  Scoring: o You can take the sum or the mean  The z-scores and population distribution will remain the same o “Most people are happy”  63% of high school/college students scored above 20  72% of disabled adults  4 is the midpoint on the 7-point scale o Disadvantages to using this approach?  Operational and Conceptual Variables:  Intelligence  Anxiety  Depression  Marital satisfaction  Else? o Remember that most conceptual definitions can be operationalized in multiple ways with multiple components Ways to Measure Variables Three Common Ways to Measure: 1. Self-report 2. Observational 3. Physiological Self- Report Measure  Def. Operationalizes a variable by recording people’s answers to questions about themselves in a questionnaire or interview  Participants report on their own behavior or state of mind o Attitudes, liking for someone, judgments about someone’s personality characteristics, intended behaviors, emotional states, etc.  Pros: o Popular and easy to use  Often use rating scales with descriptive anchors (5-point scale on life satisfaction)  Cons: o You cannot be sure that a participant is telling you the truth (biased, subjective) when using a self-report measure  Diener’s 5-point scale and the Ladder of Life Question on life satisfaction o “In the last month, how often have you felt nervous or stressed?” o Or, “in the last year, how many of the following events have occurred in your life: moving, job change, marriage, divorce, death in the family?” o Kids: self-report can be replaced (or augmented) by parent or teacher reports Observational Measures  “Behavioral Measures”  Def. operationalizes a variable by recording observable behaviors or physical traces of behaviors  Two types of behavioral measures: o Frequency:  Count of the number of behaviors that occur in a given period of time (# social interactions of kids on playground)  Number of Errors = another way to use frequency o Latency:  The amount of time it takes for a behavior to occur (reaction time; how long before kids on playground begin to interact)  Examples: o Happiness  Record how many times a day someone smiles o Intelligence  IQ test o Stress:  Observable behaviors: negative facial expressions, bodily agitation  Physical traces: tooth marks on a pencil  Stressful events: marriage, divorce, move, etc.  Con: o Observing behaviors may not allow you to determine the underlying cause for behavior Physiological Measures  Def. Operationalizes a variable by recording biological data such as: o Brain activity (fMRI, EEG, etc.) o Hormone level o Heart rate o Blood pressure o Skin conductance o Facial electromyogram  Psychological measures usually require the use of equipment to amplify, record, and analyze biological date  Brain Scans: o Structural Scans  CT Scan  MRI Scan o Functional Scans  PET Scan  fMRI Scan o EEG o And others… Scales of Measurement 1. Nominal o Attributes are only named o Weakest 2. Ordinal o Attributes can be ordered 3. Interval o Distance is meaningful 4. Ratio o Absolute zero  Categorical vs. Quantitative Variables: o Categorical:  “Nominal variables”  Events are assigned to categories  Example  Sometimes you’ll use need to numbers to code your data: Ex: Male = 0, Female = 1  Name only o Quantitative:  These levels are codes with meaningful numbers  Example: height and weight, IQ, and amount of salivary cortisol  Three Types: 1. Ordinal - Order 2. Interval - Order + equal spacing 3. Ratio - Order + equal spacing + absolute zero  Three Types of Quantitative Variables: 1. Ordinal Scale:  Permits events or scores to be ranked ordered  Applies when the numerals of quantitate variable represent a ranked order  Examples: o First, second, third place - Like in a race - Or high school GPA rankings o 4-star rating system for hotels - Combination of nominal and ordinal 2. Interval Scale:  Permits rank ordering of events with the assumption of equal intervals between adjacent events  Applies to the numerals of a quantitative variable that meet two conditions:  First, the numerals represent equal intervals (distances) between levels  Second, there is no “true zero”  Examples:  Like temperature, or your score on the IQ  No true zero - Temperature still exists even when it is 0º - A score of 0 doesn’t mean no verbal ability (for example) …but maybe very low verbal ability - Because there is no true zero, you can’t say things like “twice as hot” or “half as intelligent” 3. Ratio Scale:  Permits rank ordering of events (like ordinal) with the assumption of equal intervals between adjacent events (like interval) and a true zero point.  Examples:  Physical measurement, like intensity of light, volume of water, speed, or action potentials per second  Other measures such as income  Can make ratio comparisons like “twice as much” or “half as much” Reliability of Measurement: Are the Scores Consistent? How do I know if a study’s measurements are good measurements?  How do I know if a study has construct validity? o Validity: is it measuring what it’s supposed to measure?  Accuracy o Reliability: how consistent is the measurement  Consistency Three types of Reliability 1. Test-retest reliability o Researcher gets consistent scores every time uses the measure 2. Interrater reliability o Consistent scores are obtained no matter who measures or observes 3. Internal reliability o “Internal consistency” o A study participant gives a consistent pattern of answers, no matter how the research has phrased the question 1. Test-retest reliability  Repeat a measurement over time  If two test are high correlated, yielding same scores for most people, then the resulting measure is said to have high test-retest reliability  Example. Ash’s scores on a creative test are similar at the end and beginning of the semester 2. Internal consistency reliability  To examine the relationships among the items themselves at a single point in time  If the two tests are highly correlated, yielding similar scores for most people, the resulting measure is said to have high internal consistency reliability  It is internal because it is assessed within itself  Psychologists constructing various measures assume that all items on a scale are measuring the same characteristics  if are they should be positively correlated with one another 3. Inter-rater reliability  Observer-based personality data  Obtain measurement from multiple observers  When different observers agree with one another  high inter-rater reliability Test- Retest Reliability  Test/inventory is given twice… scores from the two tests are compared o Greater the similarity  higher the reliability.  A similar procedure: Split-half technique o Test/inventory is divided into two halves/subtests and administered to the same individuals on different occasions.  Not separated by 1st-half then 2nd half…instead might separate into 2 halves based on even-odd numbered items Inter-rater Reliability  Aka interobserver reliability  Two or more independent observers should come up with the same (or very similar) findings  Observer-based personality data o Obtain measurement from multiple observers  When different observers agree with one another  high inter-rater reliability  Very useful for naturalistic observation o Children’s behavior at preschool or park, glancing behavior at crosswalks or ATMs, etc  Why is this important? o One observer may miss or overlook a bit of behavior. o There may be some disagreement concerning exactly what was seen and how it should be rated or categorized. Internal Reliability  Consistency in a pattern of answers, no matter how a question is phrased  Important for self-report measures that contain more than one question to measure the same construct  To examine the relationships among the items themselves at a single point in time  If the two tests are highly correlated, yielding similar scores for most people, the resulting measure is said to have high internal consistency reliability  It is internal because it is assessed within itself  Internal reliability & Chronbach’s alpha o Collect data from large participant pool  Many items address same question but are worded differently (like we saw for life satisfaction) o Compute all possible correlations between items  1 correlated with 2? 1 with 3? 2 with 3? Etc. o Chronbach’s alpha is based on average of correlations and number of items o Chronbach’s alpha can be between 0-1  Higher means greater internal reliability (generally want >0.70 for selfreport measures)  If Chronbach’s alpha is low, items need to be revised before becoming part of a single scale. Using the Correlation Coefficient r to Evaluate Reliability  Correlation Coefficient o The relationship between variables is indicated by a correlation coefficient o “r” represents the degree of correlation between two variables  Both the X variable and the Y variable are interval or ratio scale measurements  Categorical and ordinal data won’t work  The data appear to be linear  Don’t use correlations with curvilinear data o Def. Statistic used to represent the relationship between variables, denoted by the symbol r o Falls between −1.00 and +1.00 o Positive correlation coefficient  As values of the independent variable increase, values of the dependent variable also increase  example between stress and depression: people who report more stressors have a higher level of depression o Negative correlation coefficient  As values of the independent variable increase, values of the dependent variable decrease  example between stress and depression: people who report more stressors have a lower level of depression  Statistical Significance o Def. Index of how likely a result is due to chance o P < .05  Probability (p) is less than 5 in 100 that the result occurred by chance alone o Statistical and clinical significance are not necessarily equivalent  Correlation Versus Causation o High correlation between an independent and a dependent variable does not mean that the independent variable caused the dependent variable o Third variable problem: Possibility that variables not measured in a study are the real cause of the relationship between the variables that are measured