Chapter 7: Qualitative & Quantitative Measurement

Chapter 7: Qualitative & Quantitative Measurement I. II. Measurement a. Is a highly developed sub-field in quantitative research design in which a researcher must make assumptions regarding the nature of different types of phenomena. Measuring is arguably the most important step towards collecting data that accurately represents the phenomena under investigation. It is almost never straight forward and usually requires a number of experts to determine which elements should be included, what instrument(s) should be used, who should do the measuring, etc. b. Why Measure? i. Measurement extends our senses. ii. Makes visible what is normally invisible. iii. Enables us to make comparisons and assumptions. c. Parts of the Measurement Process i. Conceptualization 1. Is the process of taking an abstract construct and refining it by giving it a conceptual or theoretical definition. a. Rules to Conceptualize i. Unfortunately, there is no magical way to do this. A single construct may have numerous definitions and people may agree or disagree. Typically, conceptual definitions are linked to theoretical frameworks and to value positions. Some constructs may be simple or complex; others may be highly abstract or concrete. Some may contain numerous concepts within it, sill others could stand alone. ii. Operationalization 1. Is the process of linking a conceptual definition to a specific set of measurement techniques or procedures. The specific procedures are the constructs operational definition (e.g., survey questionnaire, a method of observing events in a field setting, etc.). This process is critical because it is the bridge between the abstract theoretical domain and our observable reality. Quantitative Conceptualization and Operationalization: One More Glance a. The diagram below illustrates the measurement process for two variables that are linked together at the theoretical and empirical levels. There are three levels to consider: i. Theoretical 1. At the most abstract level, the researcher is interested in the causal relationship between two constructs, or a proposition. ii. Operational 1. At the level of operational definitions, the researcher is interested in testing a hypothesis to determine the degree of association between variables. iii. Empirical. 1. The third level is the concrete empirical world. If the operational indicators of variables are logically linked to a construct, they will capture what happens in the empirical world and relate it to the conceptual level. The measurement process links together the three levels, moving deductively from the abstract to the concrete. 17 III. Reliability a. Defined as dependability or consistency. It suggests that the same thing is repeated or reoccurs under the identical or very similar conditions. i. Types of Reliability 1. Stability Reliability a. Is reliability across time. It addresses the question: Does the measure deliver the same answer when applied in different time periods? i. Techniques to Measure Stability Reliability 1. Test-Retest Method a. A process in which a researcher continually tests and retests or readministers the indicator to the same group of people across time. 2. Representative Reliability a. Reliability across subpopulations or groups of people. It addresses the question: Does the indicator deliver the same answer when applied to different groups? i. Techniques to Measure Representative Reliability 1. Subpopulation Analysis a. Involves comparing the indicator across different subpopulations or subgroups and uses independent knowledge about subpopulations to compare. 3. Equivalence Reliability a. Applies when a researcher uses multiple indicators (e.g., several items in a questionnaire). It addresses the question: Does the measure yield consistent results across different indicators? i. Techniques to Measure Equivalence Reliability 1. Split-Half Method a. Involves dividing the indicator of the same construct into two groups, usually by a random process, and 18 2. determining whether both halves yield similar results. Inter-Coder Reliability a. Arises when there are several observers, raters, or coders of information. In a sense, each person who is observing is an indicator. A measure is reliable if the observers, raters, or coders agree with each other on a consistent basis. ii. How Do You Improve Reliability? 1. Clearly Conceptualize a. This means developing unambiguous, clear theoretical definitions. 2. Use a Precise Level of Measurement a. Indicators at higher or more precise levels (interval, ratio) of measurement are more likely to be reliable than less precise measures because the latter reveal less information. 3. Use Multiple Indicators a. This allows researchers to take measurement from a wider range of the content of a conceptual definition. 4. Use Pilot Tests a. Enables researchers to workout or identify possible mistakes in the research process. By pilot testing, a researcher is able to identify potential pitfalls or mishaps within the research process and latter go back and make efforts to fix them before reaching the hypothesis-testing phase. IV. Validity a. Refers to how well the conceptual and operational definition mesh with each other. In other words, it addresses the question: Is the researcher actually measuring what is intended. Validity is very difficult to achieve. In fact, researchers cannot have absolute confidence about validity. The reason researchers can never have absolute confidence is due to the abstract nature of the constructs social scientists often attempt to measure. i. Types of Validity 1. Face Validity a. It is a judgment by the scientific community that the indicator really measures the construct. It addresses the question: On the face of it, do people believe that the definition and method of measurement fit? 2. Content Validity a. Is a special type of face validity. It addresses the question: Is the full content of a definition represented in a measure? i. Steps in Achieving Content Validity 1. Specify the content in a construct’s definition. 2. Sample from all areas of the definition. 3. Develop an indicator that taps all of the parts of the definition. 3. Criterion Validity a. States that the validity of an indicator is verified by comparing it with another measure of the same construct in which a researcher already has confidence. i. Subtypes of Criterion Validity 1. Concurrent Validity a. An indicator must be associated with a preexisting indicator that is judged to be valid. In other words, if a person scores high on an old 19 2. 4. V. measure, they should score similarly on the new one, and vice versa. Predictive Validity a. An indicator predicts future events that are logically related to a construct. In other words, if SAT scores are supposed to predict success rates in college, a student who performs well on the SAT’s should likewise be successful in college, and vice versa. Construct Validity a. Is used for measures with multiple indicators. It addresses the question: If the measure is valid, do the various indicators operate in a consistent manner? i. Subtypes of Construct Validity 1. Convergent Validity a. Means that multiple measures of the same construct hang together or operate in similar ways. 2. Divergent Validity a. Means that multiple measures from opposite constructs should diverge or be negatively associated. 5. Internal Validity a. Means there are no errors internal to the design of the research project. It is used primarily in experimental research to talk about the possibility that an alternative explanation (e.g., a variable not controlled for in the environment) may be the actual cause of the research results. 6. External Validity a. It is the ability to generalize findings from a specific setting and small group to a broad range of settings and people. It addresses the question: If something happens in a laboratory or among a particular group of subjects, can the findings be generalized to the “real” (non-laboratory) world or to the general public. It is used primarily in experimental research. 7. Statistical Validity a. It addresses the question: Did researcher use the correct statistical procedure(s) and were its assumptions fully met. A Guide to Quantitative Measurement a. Continuous and Discrete Variables i. Continuous Variables (e.g., Interval, Ratio) 1. A variable that has a theoretically infinite number of values or attributes that flow along a continuum. ii. Discrete Variables (e.g., Nominal, Ordinal) 1. A variable that has a fixed set of separate values or attributes. b. Levels of Measurement i. The four levels of measurement categorize the degree of precision of measurement. Nominal variables provide the lowest degree of precision while ratio is at the highest. How do I know which level to use? The appropriate level of measurement for a variable depends on two things: 1. How an abstract construct is conceptualized. 2. The type of indicator or measurement that a researcher uses. ii. Nominal (lowest degree of precision) 1. Indicate only that there is a difference among categories (e.g., sex: male, female; religion: Protestant, Catholic, Jew). 20 c. iii. Ordinal 1. Measures indicate a difference, plus the categories can be ordered or ranked (e.g., letter grades: A,B,C,D,F). iv. Interval 1. Measures everything the first two do, plus it can specify the amount of distance between categories. In all cases, a variable at the interval level must have intervals of equal distance. Arbitrary zeros may also be used. v. Ratio 1. Measures do everything all the other levels do, plus there is a true zero (in other words, zero means the absence of the thing being measured), which makes it possible to state relations in terms of proportions and ratios (e.g., income: $10, $100). Special Note Regarding Levels of Measurement 1. A measure can be collapsed from a higher to a lower level of precision (e.g., ratio to ordinal) but never from a lower to a higher level of precision (e.g., nominal to ratio). 21