University:
California State University, Northridge
Course:
PSY 427 | Introduction to Psychological Testing
Academic year:

2024
Views:

419

Pages:

28
Author:

Drefddyndq9

Part 2 Spearman Brown Formula  Estimates the reliability for the entire test based on the split-half  Can also be used to estimate the affect changing the number of items on a test has on the reliability j (r ) r = 1 + ( j − 1)r * Where r* is the estimated reliability, r is the correlation between the halves, j is the new length proportional to the old length Spearman Brown Formula  For a split-half it would be 2(r ) r = (1 + r ) *  Since the full length of the test is twice the length of each half Spearman Brown Formula  Example 1: a 30 item test with a split half reliability of .65 r  The .65 * 2(.65) = .79 (1 + .65) .79 is a much better reliability than the Spearman Brown Formula  Example 2: a 30 item test with a test retest reliability of .65 is lengthened to 90 items 3(.65) 1.95 * = r = 1 + (3 − 1).65 = .85 2.3  Example 3: a 30 item test with a test retest reliability of .65 is cut to 15 items .5(.65) .325 = r = = .48 1 + (.5 − 1).65 .675 * Detour 1: Variance Sum Law Often multiple items are combined in order to create a composite score  The variance of the composite is a combination of the variances and covariances of the items creating it  General Variance Sum Law states that if X and Y are random variables:  σ 2 X ±Y = σ + σ ± 2σ XY 2 X 2 Y Detour 1: Variance Sum Law Given multiple variables we can create a variance/covariance matrix  For 3 items:  X1 X2 X3 X1 σ σ 12 σ 13 2 X 2 σ 21 σ 2 σ 23 2 X 3 σ 31 σ 32 σ 3 2 1 Detour 1: Variance Sum Law Example Variables X, Y and Z  Covariance Matrix:  X Y Z  σ X 55.83 29.52 30.33 Y 29.52 17.49 16.15 Z 30.33 16.15 29.06 By the variance sum law the composite variance would be: 2 X +Y + Z =σ 2 Total = σ + σ + σ + 2σ XY + 2σ XZ + 2σ YZ 2 X 2 Y 2 Z Detour 1: Variance Sum Law X Y Z  X 55.83 29.52 30.33 Y 29.52 17.49 16.15 Z 30.33 16.15 29.06 By the variance sum law the composite variance would be: 2 Total s = 55.83 + 17.49 + 29.06 + +2(29.52) + 2(30.33) + 2(16.15) = = 254.41 CTT: Internal Consistency Reliability If items are measuring the same construct they should elicit similar if not identical responses  Coefficient OR Cronbach’s Alpha is a widely used measure of internal consistency for continuous data  Knowing the a composite is a sum of the variances and covariances of a measure we can assess consistency by how much covariance exists between the items relative to the total variance  CTT: Internal Consistency Reliability  Coefficient Alpha k  ∑ sij α=  2 k − 1  sTotal is defined as:    2 s  Total is the composite variance (if items were summed)  sij is covariance between the ith and jth items where i is not equal to j  k is the number of items CTT: Internal Consistency Reliability  Using the same continuous items X, Y and Z  The covariance matrix is: X Y Z  The X 55.83 29.52 30.33 Y 29.52 17.49 16.15 Z 30.33 16.15 29.06 total variance is 254.41  The sum of all the covariances is 152.03 3  152.03  k  ∑ sij  α = =  2    .8964 k − 1  sTotal  3 − 1  254.41  CTT: Internal Consistency Reliability  Coefficient Alpha can also be defined as: 2 2  k sTotal − ∑ si  α=   2 k −1 sTotal  2 s  Total is the composite variance (if items were summed) 2  si is variance for each item  k is the number of items CTT: Internal Consistency Reliability  Using the same continuous items X, Y and Z  The covariance matrix is: X Y Z X 55.83 29.52 30.33 Y 29.52 17.49 16.15 Z 30.33 16.15 29.06  The total variance is 254.41  The sum of all the variances is 102.38 α 2 2  3  254.41 − 102.38  k sTotal − ∑ si  = =     .8964 2 254.41 k −1  sTotal   3 −1  CTT: Internal Consistency Reliability  From SPSS ****** Method 1 (space saver) will be used for this analysis ****** R E L I A B I L I T Y A N A L Y S I S - S C A L E (A L P H A) Reliability Coefficients N of Cases = Alpha = 100.0 .8964 N of Items = 3 CTT: Internal Consistency Reliability  Coefficient Alpha is considered a lowerbound estimate of the reliability of continuous items  It was developed by Cronbach in the 50’s but is based on an earlier formula by Kuder and Richardson in the 30’s that tackled internal consistency for dichotomous (yes/no, right/wrong) items Detour 2: Dichotomous Items  If Y is a dichotomous item: P = proportion of successes OR items answer correctly Q = proportion of failures OR items answer incorrectly   Y = P, observed proportion of successes 2 sY = PQ CTT: Internal Consistency Reliability  Kuder and Richardson developed the KR20 that is defined as 2  k sTotal − ∑ pq  α=   2 k −1 sTotal   Where pq is the variance for each dichotomous item  The KR21 is a quick and dirty estimate of the KR20 CTT: Reliability of Observations What if you’re not using a test but instead observing individual’s behaviors as a psychological assessment tool?  How can we tell if the judge’s (assessor’s) are reliable?  CTT: Reliability of Observations Typically a set of criteria are established for judging the behavior and the judge is trained on the criteria  Then to establish the reliability of both the set of criteria and the judge, multiple judges rate the same series of behaviors  The correlation between the judges is the typical measure of reliability  But, couldn’t they agree by accident? Especially on dichotomous or ordinal scales?  CTT: Reliability of Observations Kappa is a measure of inter-rater reliability that controls for chance agreement  Values range from -1 (less agreement than expected by chance) to +1 (perfect agreement)  +.75 “excellent”  .40 - .75 “fair to good”  Below .40 “poor”  Standard Error of Measurement So far we’ve talked about the standard error of measurement as the error associated with trying to estimate a true score from a specific test  This error can come from many sources  We can calculate it’s size by:  smeas = s 1− r  s is the standard deviation; r is reliability Standard Error of Measurement  Using the same continuous items X, Y and Z  The total variance is 254.41  s = SQRT(254.41) = 15.95  α = .8964 smeas = 15.95* 1 − .8964 = 5.13 CTT: The Prophecy Formula How much reliability do we want?  Typically we want values above .80  What if we don’t have them?  The Spearman-Brown can be algebraically manipulated to achieve  rd (1 − ro ) j= ro (1 − rd )  j = # of tests at the current length, rd = desired reliability, ro = observed reliability CTT: The Prophecy Formula Using the same continuous items X, Y and Z  α = .8964   What if we want a .95 reliability? rd (1 − ro ) .95(1 − .8964) .0984 = = = 2.2 ro (1 − rd ) .8964(1 − .95) .0448 j We need a test that is 2.2 times longer than the original  Nearly 7 items to achieve .95 reliability  CTT: Attenuation  Correlations are typically sought at the true score level but the presence of measurement error can cloud (attenuate) the size the relationship  We can correct the size of a correlation for the low reliability of the items.  Called the Correction for Attenuation CTT: Attenuation  Correction for attenuation is calculated as: r12 rˆ12 = r11r22    r̂12 is the corrected correlation r12 is the uncorrected correlation r11 and r22 the reliabilities of the tests CTT: Attenuation  For example X and Y are correlated at .45, X has a reliability of .8 and Y has a reliability of .6, the corrected correlation is = rˆ12 r12 = r11r22 .45 = .8*.6 .45 = .65 .48

Classical Test Theory and Reliability, Part 2

Related Documents

Recommended Documents

Description

Related Documents

Free up your schedule!

Take 5 seconds to unlock