Classical Test Theory and Reliability

Basics of Classical Test Theory Theory and Assumptions Types of Reliability
Example

Classical Test Theory
Classical Test Theory (CTT) – often called the “true score model”
Called classic relative to Item Response Theory (IRT) which is a more modern approach
CTT describes a set of psychometric procedures used to test items and scales reliability, difficulty, discrimination, etc. 1 Classical Test Theory
CTT analyses are the easiest and most widely used form of analyses. The statistics can be computed by readily available statistical packages (or even by hand)
CTT Analyses are performed on the test as a whole rather than on the item and although item statistics can be generated, they apply only to that group of students on that collection of items Classical Test Theory
Assumes that every person has a true score on an item or a scale if we can only measure it directly without error
CTT analyses assumes that a person’s test score is comprised of their “true” score plus some measurement error.
This is the common true score model X =T + E Classical Test Theory
Based on the expected values of each component for each person we can see that ε ( X i ) = ti Ei = X i − ti ε ( X i − ti ) = ε ( X i ) − ε (ti ) = ti − ti = 0

E and X are random variables, t is constant However this is theoretical and not done at the individual level. 2 Classical Test Theory
If we assume that people are randomly selected then t becomes a random variable as well and we get: X =T + E
Therefore, in CTT we assume that the error :  Is normally distributed  Uncorrelated with true score  Has a mean of Zero Without σ meas T With σ meas X=T+E True Scores T1 T2 T3  Measurement error around a T can be large or small 3 Domain Sampling Theory Another Central Component of CTT Another way of thinking about populations and samples
Domain - Population or universe of all possible items measuring a single concept or trait (theoretically infinite)
Test – a sample of items from that universe

Domain Sampling Theory A person’s true score would be obtained by having them respond to all items in the “universe” of items
We only see responses to the sample of items on the test
So, reliability is the proportion of variance in the “universe” explained by the test variance
Domain Sampling Theory A universe is made up of a (possibly infinitely) large number of items
So, as tests get longer they represent the domain better, therefore longer tests should have higher reliability
Also, if we take multiple random samples from the population we can have a distribution of sample scores that represent the population
4 Domain Sampling Theory Each random sample from the universe would be “randomly parallel” to each other
Unbiased estimate of reliability
r1t = r1 j r  1t = correlation between test and true score r  1 j = average correlation between the test and all other randomly parallel tests Classical Test Theory Reliability
Reliability is theoretically the correlation between a test-score and the true score, squared
Essentially the proportion of X that is T 2 = ρ XT σ T2 σ T2 = σ X2 σ T2 + σ E2
This can’t be measured directly so we use other methods to estimate CTT: Reliability Index
Reliability can be viewed as a measure of consistency or how well as test “holds together”
Reliability is measured on a scale of 0-1. The greater the number the higher the reliability. 5 CTT: Reliability Index
The approach to estimating reliability depends on  Estimation of “true” score  Source of measurement error
Types of reliability  Test-retest  Parallel Forms  Split-half  Internal Consistency CTT: Test-Retest Reliability
Evaluates the error associated with administering a test at two different times.
Time Sampling Error
How-To:  Give test at Time 1  Give SAME TEST at Time 2  Calculate r for the two scores
• Easy to do; one test does it all. CTT: Test-Retest Reliability
Assume 2 administrations X1 and X2 σ E2 = σ E2 ε ( X 1i ) = ε ( X 2i ) ∴ ρ X1 X 2 = 1i 2i 2 T 2 X σX X σ = = ρ XT σXσX σ 1 1 2 2
The correlation between the 2 administrations is the reliability 6 CTT: Test-Retest Reliability
Sources of error  random fluctuations in performance  uncontrolled testing conditions ○ extreme changes in weather ○ sudden noises / chronic noise ○ other distractions  internal factors ○ illness, fatigue, emotional strain, worry ○ recent experiences CTT: Test-Retest Reliability
Generally used to evaluate constant traits.
Not appropriate for qualities that change rapidly over time.
Problem: Carryover Effects  Intelligence, personality  Mood, hunger  Exposure to the test at time #1 influences scores on the test at time #2 Only a problem when the effects are random.
If everybody goes up 5pts, you still have the same variability
CTT: Test-Retest Reliability
Practice effects  Type of carryover effect  Some skills improve with practice ○ Manual dexterity, ingenuity or creativity  Practice effects may not benefit everybody in the same way. Carryover & Practice effects more of a problem with short inter-test intervals (ITI).
But, longer ITI’s have other problems
 developmental change, maturation, exposure to historical events 7 CTT: Parallel Forms Reliability Evaluates the error associated with selecting a particular set of items.
Item Sampling Error
How To:
 Develop a large pool of items (i.e. Domain) of varying difficulty.  Choose equal distributions of difficult / easy items to produce multiple forms of the same test.  Give both forms close in time.  Calculate r for the two administrations. CTT: Parallel Forms Reliability
Also Known As:  Alternative Forms or Equivalent Forms Can give parallel forms at different points in time; produces error estimates of time and item sampling.
One of the most rigorous assessments of reliability currently in use.
Infrequently used in practice – too expensive to develop two tests.
CTT: Parallel Forms Reliability
Assume 2 parallel tests X and X’ ε ( X i ) = ε ( X i' ) σ E2 = σ E2 i ' i σ XX ' σ T2 ∴ ρ XX ' = = = ρ XT σ X σ X ' σ X2
The correlation between the 2 parallel forms is the reliability 8 CTT: Split Half Reliability What if we treat halves of one test as parallel forms? (Single test as whole domain)
That’s what a split-half reliability does
This is testing for Internal Consistency
 Scores on one half of a test are correlated with scores on the second half of a test.
Big question: “How to split?”  First half vs. last half  Odd vs Even  Create item groups called testlets CTT: Split Half Reliability
How to:  Compute scores for two halves of single test, calculate r.
Problem:  Considering the domain sampling theory what’s wrong with this approach?  A 20 item test cut in half, is 2 10-item tests, what does that do to the reliability?  If only we could correct for that… Spearman Brown Formula
Estimates the reliability for the entire test based on the split-half
Can also be used to estimate the affect changing the number of items on a test has on the reliability r* is the estimated j (r ) Where reliability, r is the correlation r = between the halves, j is the 1 + ( j − 1)r new length proportional to * the old length 9 Spearman Brown Formula
For a split-half it would be r* = 2(r ) (1 + r )
Since the full length of the test is twice the length of each half Spearman Brown Formula
Example 1: a 30 item test with a split half reliability of .65 r* =
The 2(.65) = .79 (1 + .65) .79 is a much better reliability than the .65 Spearman Brown Formula
Example 2: a 30 item test with a test retest reliability of .65 is lengthened to 90 items 3(.65) 1.95 r* = 1 + (3 − 1).65 = 2.3 = .85
Example 3: a 30 item test with a test retest reliability of .65 is cut to 15 items r* = .5(.65) .325 = = .48 1 + (.5 − 1).65 .675 10 Detour 1: Variance Sum Law Often multiple items are combined in order to create a composite score
The variance of the composite is a combination of the variances and covariances of the items creating it
General Variance Sum Law states that if X and Y are random variables:
σ X2 ±Y = σ X2 + σ Y2 ± 2σ XY Detour 1: Variance Sum Law Given multiple variables we can create a variance/covariance matrix
For 3 items:
X1 X1 X2 X3 X2 X3 σ 12 σ 12 σ 13 σ 21 σ 22 σ 23 σ 31 σ 32 σ 32 Detour 1: Variance Sum Law

Example Variables X, Y and Z Covariance Matrix: X Y Z
X 55.83 29.52 30.33 Y 29.52 17.49 16.15 Z 30.33 16.15 29.06 By the variance sum law the composite variance would be: 2 σ X2 +Y + Z = σ Total = σ X2 + σ Y2 + σ Z2 + 2σ XY + 2σ XZ + 2σ YZ 11 Detour 1: Variance Sum Law X Y Z
X 55.83 29.52 30.33 Y 29.52 17.49 16.15 Z 30.33 16.15 29.06 By the variance sum law the composite variance would be: 2 sTotal = 55.83 + 17.49 + 29.06 + +2(29.52) + 2(30.33) + 2(16.15) = = 254.41 CTT: Internal Consistency Reliability If items are measuring the same construct they should elicit similar if not identical responses
Coefficient OR Cronbach’s Alpha is a widely used measure of internal consistency for continuous data
Knowing the a composite is a sum of the variances and covariances of a measure we can assess consistency by how much covariance exists between the items relative to the total variance
CTT: Internal Consistency Reliability
Coefficient Alpha α= k  ∑ sij  2 k − 1  sTotal is defined as:    2  sTotal is the composite variance (if items were summed) is covariance between the ith and jth items where i is not equal to j  k is the number of items  sij 12 CTT: Internal Consistency Reliability
Using
The the same continuous items X, Y and Z covariance matrix is: X 55.83 29.52 30.33 X Y Z Y 29.52 17.49 16.15 Z 30.33 16.15 29.06
The total variance is 254.41 sum of all the covariances is 152.03 3  152.03  k  ∑ sij  α=  2  =   = .8964 k − 1  sTotal 3 − 1  254.41  
The CTT: Internal Consistency Reliability
Coefficient Alpha α= can also be defined as: 2 2 k  sTotal − ∑ si    2 k −1 sTotal  2  sTotal is the composite variance (if items were summed) 2  si is variance for each item  k is the number of items CTT: Internal Consistency Reliability
Using
The X Y Z the same continuous items X, Y and Z covariance matrix is: X 55.83 29.52 30.33 Y 29.52 17.49 16.15 Z 30.33 16.15 29.06
The total variance is 254.41
The sum of all the variances is 102.38 α= 2 2 k  sTotal − ∑ si  3  254.41 − 102.38    =   = .8964 2 k −1  sTotal 3 −1 254.41   13 CTT: Internal Consistency Reliability
From SPSS ****** Method 1 (space saver) will be used for this analysis ****** R E L I A B I L I T Y A N A L Y S I S - S C A L E (A L P H A) Reliability Coefficients N of Cases = Alpha = 100.0 N of Items = 3 .8964 CTT: Internal Consistency Reliability
Coefficient Alpha is considered a lowerbound estimate of the reliability of continuous items
It was developed by Cronbach in the 50’s but is based on an earlier formula by Kuder and Richardson in the 30’s that tackled internal consistency for dichotomous (yes/no, right/wrong) items Detour 2: Dichotomous Items
If Y is a dichotomous item:
P = proportion of successes OR items answer correctly
Q = proportion of failures OR items answer incorrectly

Y = P, observed proportion of successes sY2 = PQ 14 CTT: Internal Consistency Reliability
Kuder and Richardson developed the KR20 that is defined as α= 2 k  sTotal − ∑ pq    2 k − 1  sTotal   Where pq is the variance for each dichotomous item KR21 is a quick and dirty estimate of the KR20
The CTT: Reliability of Observations What if you’re not using a test but instead observing individual’s behaviors as a psychological assessment tool?
How can we tell if the judge’s (assessor’s) are reliable?
CTT: Reliability of Observations Typically a set of criteria are established for judging the behavior and the judge is trained on the criteria
Then to establish the reliability of both the set of criteria and the judge, multiple judges rate the same series of behaviors
The correlation between the judges is the typical measure of reliability
But, couldn’t they agree by accident? Especially on dichotomous or ordinal scales?
15 CTT: Reliability of Observations Kappa is a measure of inter-rater reliability that controls for chance agreement
Values range from -1 (less agreement than expected by chance) to +1 (perfect agreement)
+.75 “excellent”
.40 - .75 “fair to good”
Below .40 “poor”
Standard Error of Measurement So far we’ve talked about the standard error of measurement as the error associated with trying to estimate a true score from a specific test
This error can come from many sources
We can calculate it’s size by:
smeas = s 1 − r
s is the standard deviation; r is reliability Standard Error of Measurement
Using the same continuous items X, Y and Z total variance is 254.41
s = SQRT(254.41) = 15.95
α = .8964
The smeas = 15.95* 1 − .8964 = 5.13 16 CTT: The Prophecy Formula How much reliability do we want? Typically we want values above .80
What if we don’t have them?
The Spearman-Brown can be algebraically manipulated to achieve

j= rd (1 − ro ) ro (1 − rd )  j = # of tests at the current length, rd = desired reliability, ro = observed reliability CTT: The Prophecy Formula
Using the same continuous items X, Y and Z α = .8964
What if we want a .95 reliability?
j= rd (1 − ro ) .95(1 − .8964) .0984 = = = 2.2 ro (1 − rd ) .8964(1 − .95) .0448 We need a test that is 2.2 times longer than the original
Nearly 7 items to achieve .95 reliability
CTT: Attenuation
Correlations are typically sought at the true score level but the presence of measurement error can cloud (attenuate) the size the relationship
We can correct the size of a correlation for the low reliability of the items.
Called the Correction for Attenuation 17 CTT: Attenuation
Correction for attenuation is calculated as: rˆ12 =


r12 r11r22 r̂12 is the corrected correlation r12 is the uncorrected correlation r11 and r22 the reliabilities of the tests CTT: Attenuation
For example X and Y are correlated at .45, X has a reliability of .8 and Y has a reliability of .6, the corrected correlation is rˆ12 = r12 .45 .45 = = = .65 r11r22 .8*.6 .48 18