Validity

 The extent to which a test measures what it was designed to measure.  Agreement between a test score or measure and the quality it is believed to measure.  Proliferation of definitions led to a dilution of the meaning of the word into all kinds of “validities” Internal validity – Cause and effect in experimentation; high levels of control; elimination of confounding variables  External validity - to what extent one may safely generalize the (internally valid) causal inference (a) from the sample studied to the defined target population and (b) to other populations (i.e. across time and space). Generalize to other people  Population validity – can the sample results be generalized to the target population  Ecological validity - whether the results can be applied to real life situations. Generalize to other (real) situations   Content validity – when trying to measure a domain are all sub-domains represented    When measuring depression are all 16 clinical criteria represented in the items Very complimentary to domain sampling theory and reliability However, often high levels of content validity will lead to lower internal consistency reliability  Construct validity – overall are you measuring what you are intending to measure    Intentional validity – are you measuring what you are intending and not something else. Requires that constructs be specific enough to differentiate Representation validity or translation validity – how well have the constructs been translated into measureable outcomes. Validity of the operational definitions Face validity – Does a test “appear” to be measuring the content of interest. Do questions about depression have the words “sad” or “depressed” in them  Construct    Validity Observation validity – how good are the measures themselves. Akin to reliability Convergent validity - Convergent validity refers to the degree to which a measure is correlated with other measures that it is theoretically predicted to correlate with. Discriminant validity - Discriminant validity describes the degree to which the operationalization does not correlate with other operationalizations that it theoretically should not correlated with.  Criterion-Related Validity - the success of measures used for prediction or estimation. There are two types: Concurrent validity - the degree to which a test correlates with an external criteria that is measured at the same time (e.g. does a depression inventory correlated with clinical diagnoses)  Predictive validity - the degree to which a test predicts (correlates) with an external criteria that is measured some time in the future (e.g. does a depression inventory score predict later clinical diagnosis)   Social validity – refers to the social importance and acceptability of a measure  There is a total mess of “validities” and their definitions, what to do?  1985 - Joint Committee of    AERA: American Education Research Association APA: American Psychological Association NCME: National Council on Measurement in Education  developed Standards for Educational and Psychological Testing (revised in 1999).  According to the Joint Committee:  Validity is the evidence for inferences made about a test score.  Three types of evidence:    Content-related Criterion-related Construct-related  Different from the notion of “different types of validity”  Content-related  evidence (Content Validity) Based upon an analysis of the body of knowledge surveyed.  Criterion-related Validity)  evidence (Criterion Based upon the relationship between scores on a particular test and performance or abilities on a second measure (or in real life).  Construct-related Validity)  evidence (Construct Based upon an investigation of the psychological constructs or characteristics of the test.  Face    Validity The mere appearance that a test has validity. Does the test look like it measures what it is supposed to measure? Do the items seem to be reasonably related to the perceived purpose of the test.  Does a depression inventory ask questions about being sad?   Not a “real” measure of validity, but one that is commonly seen in the literature. Not considered legitimate form of validity by the Joint Committee.  Does the test adequately sample the content or behavior domain that it is designed to measure?  If items are not a good sample, results of testing will be misleading.  Usually developed during test development.   Not generally empirically evaluated. Judgment of subject matter experts.  To develop a test with high content-related evidence of validity, you need:    good logic intuitive skills Perseverance  Must   consider: wording reading level  Other  Construct underrepresentation: failure to capture important components of a construct.   content-related evidence terms Test is designed for chapters 1-10 but only chapters 18 show up on the test. Construct-irrelevant variance: occurs when scores are influenced by factors irrelevant to the construct.  Test is well-intentioned, but problems secondary to the test negatively influence the results (e.g., reading level, vocabulary, unmeasured secondary domains)  Tells us how well a test corresponds with a particular criterion    criterion: behavioral or measurable outcome SAT predicting GPA (GPA is criterion) BDI scores predicting suicidality (suicide is criterion).  Used to “predict the future” or “predict the present.”  Predictive     Validity Evidence forecasting the future how well does a test predict future outcomes SAT predicting 1st yr GPA most tests don’t have great predictive validity  decrease due to time & method variance  Concurrent    Validity Evidence forecasting the present how well does a test predict current similar outcomes job samples, alternative tests used to demonstrate concurrent validity evidence  generally estimates higher than predictive validity  Validity    Coefficient correlation between the test and the criterion usually between .30 and .60 in real life. In general, as long as they are statistically significant, evidence is considered valid.  However,   recall that r2 indicates explained variance. SO, in reality, we are only looking at explained criterion variance in the range of 9 to 36%.  Sound Problematic??  Look for changes in the cause of relationships. (third variable effect)  E.g. Situational factors during validation that are replicated in later uses of the scale  Examine   Optimally the criterion should be something the test is trying to measure If the criterion is not valid and reliable, you have no evidence of criterion-related validity!  Review study.  what the criterion really means. the subject population in the validity If the normative sample is not representative, you have little evidence of criterion-related validity.  Ensure the sample size in the validity study was adequate.  Never confuse the criterion with the predictor.     GREs are used to predict success in grad school Some grad programs may admit low GRE students but then require a certain GRE before they can graduate. So, low GRE scores succeed, this demonstrates poor predictive validity! But the process was dumb to begin with…  Watch for restricted ranges.  Review   evidence for validity generalization. Tests only given in laboratory settings, then expected to demonstrate validity in classrooms? Ecological validity?  Consider   differential prediction. Just because a test has good predictive validity for the normative sample may not ensure good predictive validity for people outside the normative sample. External validity?  Construct: synthesis  What is Intelligence? Love? Depression?  Construct   something constructed by mental Validity Evidence assembling evidence about what a test means (and what it doesn’t) sequential process; generally takes several studies  Convergent  Evidence obtained when a measure correlates well with other tests believed to measure the same construct.  Self-report, collateral-report measures  Discriminant   Evidence obtained when a measure correlates less strong with other tests believed to measure something slightly different This does not mean any old test that you know won’t correlate; should be something that could be related but you want to show is separate  Example: IQ and Achievement Tests  Standard Error of Estimate: s= s= sy est . Y −Yˆ     N −1  (1 − r )    N −2 2 sest . standard error of estimate s y standard deviation of the test r validity of the test  Essentially, this is regression all over again.  Maximum Validity depends on Reliability r12max = r1r2 r12max is the maximum validity  r1 is the reliability of test 1  r2 is the reliability of test 1  Reliability of Test 1 0.8 0.6 0.4 0.2 0 1 0.8 0.6 0.4 0.2 0 1 0.8 0.6 0.4 0.2 0 1 0.8 0.6 0.4 0.2 0 Reliability of Criterion 1 1 1 1 1 1 0.5 0.5 0.5 0.5 0.5 0.5 0.2 0.2 0.2 0.2 0.2 0.2 0 0 0 0 0 0 Maximum Validity (Correlation) 1.00 0.89 0.77 0.63 0.45 0.00 0.71 0.63 0.55 0.45 0.32 0.00 0.45 0.40 0.35 0.28 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00