Lecture 8 - Reliability

Lecture 8 Reliability The tendency of a measurement instrument to provide the same comparative information on repeated occasions (different situations, different time frame etc.) The relation between items is used to assess how well the scale is tapping into the latent variable Consistency or stability of a measure of behavior Latent constructs -> tests are indicators of abstract constructs In order to understand this concept it is important to know that within psychology, we are often trying to assess the underlying latent constructs Latent constructs: ➢ An abstract, underlying construct that forms the basis of a behavior. ➢ Latent variable (true score) - the variable we are interested in, but it can’t be observed directly. In order to still be able to measure it, we look at indicators of this variable. These can be measured. There is a causal relationship between the latent variables and their indicators, even though it is kind of impossible to call it causal. The latent variable is the true score and the indicators depict the observed score. ➢ Examples: o IQ can’t be observed directly (latent variable), but based on research conducted over the years, we know ways in which IQ manifests itself include processing speed, vocabulary, memory, and problem solving. These factors are therefore indicate IQ. So because you are intelligent, you should score higher on processing speed, vocabulary, memory, and problem solving. o Risk assessment- The risk to recidivate is a latent variable, because we can’t observe it directly. But we can observe directly is the static and dynamic risk factors. So because you have a high risk to recidivate, you should also score higher on these risk factors. o This causal relationship implies that if the latent variables cause indicators, then there should be a relation between observed values and true scores. If there is a relationship between IQ and (e.g.) memory, then the scores of both of these variables should be related. o Each of the indicators should be correlated with each other. A measurement is an additive composite of: ➢ true ability or construct that is being measured ➢ random error Observed score = True score + Error • • • The sum of the variabilities of the true score and the error make up the variability of the observed score-> Var(O) = Var(T) + Var(E) But can we actually observe the true score and the error? ➢ True score can’t be observed. ➢ there is no way for us to observe the error ➢ This implies that reliability is: o a theoretical concept o Can’t be directly computed. o It must be estimated. o Measurement errors- An extra influence on this estimate is the errors in the real world, which can affect the consistency Measurement error ➢ The variation of a set of instruments around the true mean (also called error variance). ➢ Error is the cumulative effect of extraneous influences ➢ Causes: o Random factors ▪ transitory personal factors ▪ random environmental factors ▪ When there is no random error, there is less variation than when there is a random error; random error increases the variation. The mean is the same no matter whether or whether or not there is a random error. o Systematic factors ▪ factors that systematically affect measurement ▪ cause bias in the measurement ▪ Don’t affect the variability. However, when a systematic factor is present, the mean changes. Because the systematic error causes a change in the mean, we speak of bias. • • A reliable instrument performs in a consistent way. Scores on an instrument should only change when a latent score changes. o Example: when you are measuring depression (latent trait) then the score on an instrument should only change when the depression has decreased or increased. A perfectly reliable measurement would only measure the true score, but this is impossible. As reliability increases, error variance decreases and the observed score is closer to the true score ( also true the other way around). When the error is large, the observed score is further away from the true score, which means that the observed score is a poor estimate of the true score and the reliability is poor. Types of reliability: Reliability focuses on 3 aspects of a measure: 1. Equivalence: ➢ consistency between two versions of the same measure ➢ parallel tests 2. Stability: ➢ consistency over time + between individuals ➢ test-retest ➢ inter-rater- consistency between assessors; doesn’t matter who does the tests 3. Internal cconsistency: ➢ similarity across items – the higher the score is, the more instruments re capable of measuring this underlying latent variable ➢ Cronbach’s alpha- below 7- unacceptable • • • • Validity • • • • • • • • • the degree to which a tool measures what it claims to measure. ➢ example: o If you are administering a test to job applicants to screen them for suitability for the job, then this test is valid if the scores highly relate to the examinees’ future performance on the job. A risk assessment measure is valid if it accurately predicts reoffending validity does not refer to a test score or the items on an instrument. It refers to to the interpretation of the test scores and item scores. validity is about the accuracy or legitimacy of one’s interpretations It’s important for: ➢ Basic research ➢ Social decision making ➢ Test-based decisions Validity is a matter of degree, there is no definite threshold; different from reliability, because reliability has specific cutoff scores. The higher the score is, the more valid the interpretations are. Validity is an ongoing process. You have to continue testing how well a test predicts what you’re interested in and in what contexts. The validity of a test should be based on evidence and theory. Types of validity: ➢ Theoretical constructs- can be divided into face validity (the degree to which the test seems to measure what it is supposed to measure) and content validity (the degree to which the items of a test are measuring what they are supposed to measure). ➢ Empirical constructs - can be divided into construct validity (the degree to which the test measures what it is supposed to measure) and criterion validity (the degree to which an operationalization of a construct is predictive of or related to that construct). o Construct validity- can be divided into discriminant validity (the degree to which a test does not measure unrelated constructs) and convergent validity (the degree to which a test measures related constructs). o Criterion validity- can be divided into predictive validity (the degree to which a test predicts behavior) and concurrent validity (the degree to which a test is similar to a new test) • Reliability+validity ➢ Reliability does not indicate whether or not a particular measure is an accurate or valid measure of the variable of interest. ➢ A measure can be reliable but not valid -> a test can measure the same construct similarly across situations, persons, etc. but if it is not measuring the construct of interest, it is not valid. ➢ Issues: o Issue of a deception- poses a threat to the validity and reliability of an instrument. It can cause biases. o Biases are systematic errors that affect the average mean scores, and in that way affect the interpretation of the final results. They can: - impair the validity of a test - adversely impact the effectsize estimation. Tackling pitfalls • Counteract biases: ➢ It is helpful to, if possible, highlight the anonymity of the assessment. In risk assessment this is often difficult, because with risk assessment often an interview is conducted. ➢ It is important to minimize fatigue, stress, and distractions during the assessment. ➢ It might be helpful to make patients believe that dishonesty can be detected. o validity can be increased when participants merely believe that dishonesty can be detected o Without coercing/lying to participants! ➢ You can improve the content of your test by: o Following scale construction principles o Straightforward, simple wording o Reducing emotionally-charged words o Reverse-scoring items- you ‘force’ the patient to think about the question ➢ You can use validity scales • Example 1: MMPI ➢ Minnesota Multiphasic Personality Inventory ➢ L-scale- refers to items that measure weaknesses that everyone will admit to (e.g. “I sometimes question my actions”). If someone doesnt admit to any of these items, this shows that they are not answering truthfully ➢ F-scale- consists of items that measure low baseline events that most individuals will respond “false” to (e.g. “I’ve never told a lie”). Everyone has told a lie, so if someone responds to this that they have never did, this shows that they rae not being truthful ➢ TRIN (True Response Inconsistency)- a scale that consists of 20 reversely scored items. They are used to measure compliance. ➢ VRIN (Variable Response Inconsistency)- a scale that consists of repeated questions, but in different wording. This scale helps to detect random answers and potentially helps to indicate an individual faking bad. o Elevated scores on the VRIN-and F-scales- indicate that the person answered the questions randomly, which provides an invalid profile. o Normal VRIN scores and high F-scores- either refers to that the patient has serious psychological issues that require professional attention, or that the patient is faking bad (malingering) • Example 2: HIT ➢ How I Think questionnaire ➢ measures cognitive distortionsand ➢ consists of a subscale called “anomalous responding” ➢ This scale was developed to screen for inaccurate or suspect responding. ➢ Items include “Sometimes I gossip about other people”, “I have covered up things that I’ve done”, and “I have tried to get even with someone”. Everyone has been in situations like these at least once. So if someone is denying these statements, then you can question if they are being trutful. ➢ Scores higher than 3.5 indicate outliers and a suspicion of anomalous responding ➢ Scores higher than 4.25 are considered invalid Signal detection theory • This theory is based on the idea that the decisions we make are made in the presence of uncertainty. • It also refers to the technique that can be used to evaluate sensitivity in decision making. ➢ Decisions are made against a background of uncertainty. ➢ The goal as decision makers is to filter out the decision signal from background noise -> Both represented probabilistically within the decision-maker • Can be applied to ➢ Any binary decision making situation ➢ Where the response can be compared to the actual presence/absence of the targe ➢ graph B- the signal (reoffending) is the right graph; The left graphis the noise that can cause errors that can impair our signal detection judgment. Vertical line is the reference line. Everything on the right of the line are the hits. ➢ graph C- little red part is the overlap. This area are the false alarms. The green area in graph C refers to the correct rejection. ➢ Graph D- presents low subject sensitivity ➢ graph E- presents high subject sensitivity, meaning that there are less errors in the decision. • AUC ➢ helps us to classify the future recidivists and the non-recidivists ➢ An AUC of 1.00 represents the perfect discrimination/prediction; almost never the case. The closer to 1, the better the prediction ➢ The AUC can be calculated the ROC curve analysis. This requires 2 types of information: 1. Total risk scores and the clinical score to classify a person as low-medium-high risk 2. A dichotomous outcome (e.g., violent conviction vs. no violent conviction upon follow-up) • SDT is especially useful due to: ➢ The difficult in determining a cutoff point for diagnoses ➢ Different base rates and how they might affect diagnoses ➢ The relation between the test scores and the actual group membership (and their link to the base rate respectively) • It is useful to use ROC/AUC, but there has also been some criticism: ➢ Heterogenous samples may mask effects ➢ Laboratory experiments result in better predictions than field studies Neuroscience • • • • • • • There is a lack of insight into the different biological and cognitive factors that can cause antisocial behavior. This hampers the development of personalized treatment programs that truly fit the needs of different types of antisocial individuals. Superficially, similar behavior can have widely different causes. Being able to assess underlying factors will provide you with a more complete picture. Biocognitive fingerprint- able to take into account the full cognitive and biological make-up of an individual. ➢ This will then be more objective and measurable than many other risk factors. ➢ will also help us with understanding the motivations of offending Neurolaw-“the study of how neuroscience is, could be, or should be used in the legal system” Neurobiological factors can have an impact on offending and reoffending behavior: ➢ Acquired brain injuries ➢ Neurological disorders ➢ Genetic abnormalities ➢ These factors have often been reported in delinquent populations. ➢ We know that there is an association between these neurobiological deficits and potential offending and reoffending so it’s important whether these deficits were present at the time of the offense and whether these were related to the offense. If this is the case -> these factors should be addressed in treatment and risk management Diagnosis and treament: 1. start by collecting data about the geneticsand cognitions using neuroimaging. 2. data mining- In this phase you start to explore the latent variables that might explain variance in the population. 3. Individual fingerprint- look for individual differences and connect them to the underlying factors. If you have found these differences you can fingerprint them-> you determine the specific profile of each individual ➢ classify each individual. ➢ The dimensions in the fingerprints provide specific sets of candidate target areas for treatment. In this way, you can personalize intervention. ➢ The next step is: o incorporate the use of fingerprints into clinical practice o to improve diagnosis o try to help individuals to modify dysfunctional behavior o potentially for risk assessment. ➢ Childhood traumatic brain injury o It was found that this leads to an increased risk of offending behavior ➢ Incarcerated offenders with poor neurocognitive functioning o It was found that this population is more likely to drop out of treatment o if you are unable to tackle underlying needs, it could lead to unsuccessful treatment • Use of neurobiological information in forensic questions (Kempes et al., 2019) ➢ criminal responsibility- in less than 50% of the cases, this neurobiological information was used in the assessment of criminal responsibility ➢ risk assessment - in 65% of the cases in which the neurobiological information was used and was related to the index offense ,the information was also used in risk assessment ➢ risk management- in 43.7% of these cases, the information was also used in risk management • Pitfalls ➢ Extrapolating group-level knowledge to generate inferences about the individual (similar to actuarial assessment tools). ➢ Using with great caution, similar to risk assessment ➢ Science and law use different levels of description of behavior and explanation of human behavior ➢ Psycho-legal fallacy - the mistake of assuming that identifying a biomechanical cause by itself excuses behavior ➢ Psychiatric disorders exhibit a dimensional structure • Conclusion: ➢ Neuroscientific insights can be used to: o Uncover risks and needs in individuals o Improve responsivity in risk assessment and management ➢ However, neuroscientific insights are often not used in the explanation of offending or in treatment ➢ practical and ethical considerations that are important. ➢ deception no longer plays a role, because the individual has no influence on predetermined biological factors. ➢ It is also promising for the use in forensic clinical practice, but it is still a work in progress.