Statistics Lecture 6: Reliability and Validity

Lecture 6: Reliability and Validity Critical Values -​ A critical region, also known as the rejection region, is a set of values for the test statistic for which the null hypothesis is rejected. -​ i.e. if the observed test statistic is in the critical region then we reject the null and accept the alternative hypothesis -​ If the calculated value is > critical value = statistically significant One Tailed vs. Two Tailed Test -​ One Tailed: indicates the null should be rejected when the test value is in the critical region on one side of the mean. -​ It is either a right tailed test or left tailed test depending on the direction of the inequality of the alternative hypothesis -​ 5% region on one side -​ Two Tailed: in this the null hypothesis should be rejected when the test value is in either of the two critical regions -​ 2.5% regions on each side Statistical Significance: when there is a statistical or notable difference (but the difference may not be enough to be valuable) Clinical Significance: statistically significant and clinically useful (the difference is valuable) Feasibility -​ Is it practical? Can it be done? How easy is it to use? -​ Consider cost, time, training, or limitations of the sample Reliability (Precision) -​ Whether or not the test can consistently measure what it's supposed to -​ Consistent, precise, repeatable, reliable -​ There are 3 main factors to reliability: 1.​ Stability a.​ Consistent and enduring (does not change over time) b.​ Should have high correlation coefficient (same results over time) 2.​ Homogeneity -​ -​ -​ -​ -​ a.​ The extent to which items on a multi-item instrument are consistent with one another (eg. survey questions) b.​ Can be assessed using Cronbach’s alpha for internal consistency reliability 3.​ Equivalence a.​ How well multiple forms or users of an instrument produce the same results b.​ Must measure inter-rater reliability when using more than one researcher to assess results Equivalence reliability: the extent to which two items measure the same concepts at identical level Stability reliability: measuring the same variable over time Internal consistency: extent to which test assesses the same characteristic Inter-rater reliability: extent to which two or more individuals (raters) agree Intra-rater reliability: same assessment completed by same rater on two or more occasions Measuring Reliability -​ % agreement / Kappa statistic → measures inter-rater reliability -​ Cronbach’s alpha → correlation of items -​ 0-1 scale; >0.8 is good -​ Factor analysis -​ Test-retest Validity (Accuracy) -​ Are you measuring what you claim to measure? Is it significant? -​ Addresses the appropriateness of of the data rather than whether the measurements are repeatable -​ Types of validity: 1.​ Content Validity a.​ Face validity/content validity: depends on the judgment of the observer (peer reviewing, literature searching, considering what variables to include) 2.​ Criterion Validity (examine correlations with variables that you expect to be linked) a.​ Convergent Validity: examine correlations with existing tools/measures or instruments (do previous studies agree with your findings?) i.​ A correlation coefficient of > or = 0.4 strengthens validity b.​ Predictive Validity: after using your measurement tools can it accurately predict the intended outcome? c.​ Divergent Validity: when a tool measures the opposite variable of a previously validated tool 3.​ Construct Validity a.​ The extent to which a tool measures the intended construct -​ Discriminant validity: can it discriminate between individuals/responses? Does it only measure what it should? Reliability and Validity Connections -​ A measurement must be reliable first before it can be valid -​ A test can be reliable but that doesn’t mean it's valid!! (accurate)