University:
California State University, Northridge
Course:
PSY 427 | Introduction to Psychological Testing
Academic year:

2024
Views:

48

Pages:

25
Author:

Donazioheb

Part 1 Classical Test Theory  Classical Test Theory (CTT) – often called the “true score model”  Called classic relative to Item Response Theory (IRT) which is a more modern approach  CTT describes a set of psychometric procedures used to test items and scales reliability, difficulty, discrimination, etc. Classical Test Theory  CTT analyses are the easiest and most widely used form of analyses. The statistics can be computed by readily available statistical packages (or even by hand)  CTT Analyses are performed on the test as a whole rather than on the item and although item statistics can be generated, they apply only to that group of students on that collection of items Classical Test Theory  Assumes that every person has a true score on an item or a scale if we can only measure it directly without error  CTT analyses assumes that a person’s test score is comprised of their “true” score plus some measurement error.  This is the common true score model X= T + E Classical Test Theory  Based on the expected values of each component for each person we can see that ε ( X i ) = ti E= X i − ti i ε ( X i − ti ) = ε ( X i ) − ε (ti ) = ti − ti = 0 E and X are random variables, t is constant  However this is theoretical and not done at the individual level.  Classical Test Theory  If we assume that people are randomly selected then t becomes a random variable as well and we get: X= T + E  Therefore, error : in CTT we assume that the  Is normally distributed  Uncorrelated with true score  Has a mean of Zero Without σ meas T With σ meas X=T+E True Scores T1 T2 T3  Measurement error around a T can be large or small Domain Sampling Theory Another Central Component of CTT  Another way of thinking about populations and samples  Domain - Population or universe of all possible items measuring a single concept or trait (theoretically infinite)  Test – a sample of items from that universe  Domain Sampling Theory A person’s true score would be obtained by having them respond to all items in the “universe” of items  We only see responses to the sample of items on the test  So, reliability is the proportion of variance in the “universe” explained by the test variance  Domain Sampling Theory A universe is made up of a (possibly infinitely) large number of items  So, as tests get longer they represent the domain better, therefore longer tests should have higher reliability  Also, if we take multiple random samples from the population we can have a distribution of sample scores that represent the population  Domain Sampling Theory Each random sample from the universe would be “randomly parallel” to each other  Unbiased estimate of reliability  r1t = r1 j r  1t = correlation between test and true score r  1 j = average correlation between the test and all other randomly parallel tests Classical Test Theory Reliability  Reliability is theoretically the correlation between a test-score and the true score, squared  Essentially the proportion of X that is T σ σ ρ= = 2 2 σ σT + σ E 2 XT  This 2 T 2 X 2 T can’t be measured directly so we use other methods to estimate CTT: Reliability Index  Reliability can be viewed as a measure of consistency or how well as test “holds together”  Reliability is measured on a scale of 0-1. The greater the number the higher the reliability. CTT: Reliability Index  The approach to estimating reliability depends on  Estimation of “true” score  Source of measurement error  Types of reliability  Test-retest  Parallel Forms  Split-half  Internal Consistency CTT: Test-Retest Reliability  Evaluates the error associated with administering a test at two different times.  Time Sampling Error  How-To:  Give test at Time 1  Give SAME TEST at Time 2  Calculate r for the two scores • Easy to do; one test does it all. CTT: Test-Retest Reliability  Assume 2 administrations X1 and X2 ε ( X 1i ) = ε ( X 2i ) ∴ ρ X1 X 2 =σ 2 E2 i 2 σX X σT = = 2 =ρ XT σX σX σX 1 1  The σ 2 E1i 2 2 correlation between the 2 administrations is the reliability CTT: Test-Retest Reliability  Sources of error  random fluctuations in performance  uncontrolled testing conditions ○ extreme changes in weather ○ sudden noises / chronic noise ○ other distractions  internal factors ○ illness, fatigue, emotional strain, worry ○ recent experiences CTT: Test-Retest Reliability  Generally used to evaluate constant traits.  Intelligence, personality  Not appropriate for qualities that change rapidly over time.  Mood, hunger  Problem: Carryover Effects  Exposure to the test at time #1 influences scores on the test at time #2  Only a problem when the effects are random.  If everybody goes up 5pts, you still have the same variability CTT: Test-Retest Reliability  Practice effects  Type of carryover effect  Some skills improve with practice ○ Manual dexterity, ingenuity or creativity  Practice effects may not benefit everybody in the same way. Carryover & Practice effects more of a problem with short inter-test intervals (ITI).  But, longer ITI’s have other problems   developmental change, maturation, exposure to historical events CTT: Parallel Forms Reliability Evaluates the error associated with selecting a particular set of items.  Item Sampling Error  How To:   Develop a large pool of items (i.e. Domain) of varying difficulty.  Choose equal distributions of difficult / easy items to produce multiple forms of the same test.  Give both forms close in time.  Calculate r for the two administrations. CTT: Parallel Forms Reliability  Also Known As:  Alternative Forms or Equivalent Forms Can give parallel forms at different points in time; produces error estimates of time and item sampling.  One of the most rigorous assessments of reliability currently in use.  Infrequently used in practice – too expensive to develop two tests.  CTT: Parallel Forms Reliability  Assume 2 parallel tests X and X’ ε (Xi ) = ε (X ) ' i ∴ ρ XX '  The σ =σ 2 Ei 2 Ei' σ XX ' σ = = = ρ XT σ Xσ X ' σ 2 T 2 X correlation between the 2 parallel forms is the reliability CTT: Split Half Reliability What if we treat halves of one test as parallel forms? (Single test as whole domain)  That’s what a split-half reliability does  This is testing for Internal Consistency   Scores on one half of a test are correlated with scores on the second half of a test.  Big question: “How to split?”  First half vs. last half  Odd vs Even  Create item groups called testlets CTT: Split Half Reliability  How to:  Compute scores for two halves of single test, calculate r.  Problem:  Considering the domain sampling theory what’s wrong with this approach?  A 20 item test cut in half, is 2 10-item tests, what does that do to the reliability?  If only we could correct for that…

Classical Test Theory and Reliability, Part 1

Related Documents

Recommended Documents

Description

Related Documents

Free up your schedule!

Take 5 seconds to unlock