Nonreactive Research & Secondary Analysis

Nonreactive Research & Secondary Analysis I. II. III. Nonreactive Research a. Experiments and survey research are both reactive; that is, the people being studied are aware of that fact. The following quantitative research techniques are nonreactive; that is, those being studied are not aware that they are part of a research project. Nonreactive Measurement a. The Logic of Nonreactive Research i. The critical thing about nonreactive or unobtrusive measures (i.e., measures that are not obtrusive or intrusive) is that the people being studied are not aware of it but leave evidence of their social behavior or actions “naturally.” The observant researcher infers from the evidence to behavior or attitudes without disrupting those being studied. b. Varieties of Nonreactive or Unobtrusive Observation i. Erosion Measures 1. Where selective wear is used as a measure. ii. Accretion Measures 1. Where the measures are deposits of something left behind. c. Recording and Documentation i. Steps for Success When Using Nonreactive Approaches 1. Conceptualize a concept. 2. Link the construct to nonreactive empirical evidence. 3. Clearly operationalize the construct. 4. Collect data following clear rules and guidelines. Content Analysis a. Is a technique for gathering and analyzing the content of recordable elements in communication. The content refers to words, meanings, pictures, symbols, ideas, themes, or any message that can be communicated. The recordable elements in communication include anything written, visual, or spoken that serves as a medium of communication. b. Topics Appropriate for Content Analysis i. Problems involving large volumes of text. ii. Helpful when a topic can be studied “at a distance.” iii. When the researcher is attempting to reveal messages in a text that are difficult to see with casual observation. c. Measurement and Coding i. General Issues 1. Social researchers must carefully design and document procedures for coding and recording so that replication is possible. Constructs in content analysis are operationalized with a coding system: a set of instructions or rules on how to systematically observe and record content from text. ii. Unit of Analysis 1. The unit of analysis can vary greatly in content analysis. It can be a word, phrase, theme, plot, newspaper article, character, and so forth. iii. What is Measured? 1. Measurement in content analysis uses structured observation; systematic, careful observation based on written rules. The rules explain how to categorize and classify observations. 2. Characteristics of the Coding System a. Frequency i. Simply means counting whether or not something occurs and, if it occurs, how often. b. Direction 47 IV. i. Involves noting the direction of the messages in the content along some continuum (e.g., positive, negative, supportive, opposed). c. Intensity i. Involves the strength or power of a message in a direction. d. Space i. A researcher can record the size of a text message or the amount of space or volume allocated to it. Space in written text is measured by counting words, sentences, paragraphs, or space on a page (e.g., square inches). For video or audio text, space can be measured by the amount of time allocated. d. Coding, Validity, and Reliability i. Manifest Coding 1. Coding the visible, surface content in a text is called manifest coding a. Reliability i. Manifest coding is very reliable because the phrase or word either is or is not present b. Validity i. The same word can take on different meanings depending on the context. The possibility that there are multiple meanings of a word limits the measurement validity. ii. Latent coding 1. A researcher uses latent coding (also known as semantic analysis) and looks for the underlying, implicit meaning in the content of a text. Before determining the implicit meanings, the researcher must provide general guidelines for interpretation of the text and for determining whether particular themes or moods are present. a. Reliability i. Tends to be less reliable than manifest coding because it depends on a researcher’s knowledge of language and social meaning. b. Validity i. The validity of latent coding can exceed that of manifest coding because people communicate meaning in many implicit ways that depend on context, not just in specific words. c. Intercoder Reliability i. Reliability is usually determined using intercoder reliability, a type of equivalence reliability, with a statistical coefficient that tells the degree of consistency among coders. The coefficient is always reported with the results of content analysis research. Existing Statistics/Document and Secondary Analysis i. Topics Appropriate for Existing Statistics Research 1. Any topic on which information has been collected and is publicly available can be studied. Existing statistics research is best for topics that involve information collected by large bureaucratic organizations both public and private. ii. Social Indicators 1. Is any measure of social well being used in policy. An example of a social indicator is the FBI’s uniform crime index. It indicates the amount of crime in U.S. society. Social indicators can measure negative aspects of social life, such as infant mortality rate and so forth. iii. Locating Data 48 1. The main sources of existing statistics are government or international agencies and private sources. An enormous volume and variety of information exists. If you plan to conduct existing statistics research, it is wise to discuss your interests with an information professional- in some cases, a reference librarian, who can point you in the direction of possible resources. The single most valuable source of statistical information about the United States is the Statistical Abstract of the United States, which has been published annually (with a few exceptions) since 1878. iv. Secondary Survey Data 1. Secondary analysis is a special case of existing statistics; it is the re-analysis of previously collected survey or other data that were originally collected by others. It is relatively inexpensive; it permits comparisons across groups, nations or time; it facilitates replication; and it permits asking about issues not thought of by the original researchers. a. Questions a Researcher Interested in Secondary Research Should Ask! i. Are the secondary data appropriate for the research question? ii. What theory and hypothesis can a researcher use with the data? iii. Is the researcher already familiar with the substantive area? iv. Does the researcher understand how the data were originally gathered and coded? 2. Common Mistakes Made by Researchers Who Use Existing Statistics a. Existing statistics and secondary data are not trouble free simply because a government agency or other source gathered the original data. A common error is the fallacy of misplaced concreteness. It occurs when someone gives a false impression of accuracy by quoting statistics in greater detail than warranted by how the statistics are collected and by overloading detail. For example, in order to impress an audience of one’s command of particulars, a politician might say that the population of South Africa is 36,075,861, when they should say it is roughly 36 million. 3. Reliability a. Problems with reliability can plague existing statistics research. Listed below are some problems researchers may encounter regarding reliability: i. Stability Reliability 1. Stability reliability problems develop when official definitions or the method of collecting information changes over time. ii. Equivalence Reliability 1. Equivalence reliability can also be a problem. For example, a measure of crime across a nation depends on each police department’s providing accurate information. If departments in one region of a country have sloppy bookkeeping, the measure losses equivalence reliability. iii. Representative Reliability 1. Representative reliability can be a serious problem in official government statistics. This could result from discrimination by police, institutional racism or poor instrument 49 construction, which could lead to the misidentification of a person. 4. V. Validity a. Listed below are some problems researchers may encounter regarding validity: i. Validity problems can occur when the researcher’s theoretical definition does not match that of the government agency or organization that collected the information. Official policies and procedures specify definitions for official statistics. For example, a researcher defines a work injury as including minor cuts, bruises, and sprains that occur on the job, but the official definition in government reports only includes injuries that require a visit to a physician or hospital. ii. Another validity problem arises when official statistics are a surrogate or proxy for a construct in which a researcher is really interested. This is necessary because the researcher cannot collect original data. For example, the researcher wants to know how many people have been robbed, so he or she uses police statistics on robbery arrests as a proxy. But the measure is not entirely valid because many robberies are not reported to the police, and reported robberies do not always result in an arrest. iii. A third validity problem arises because the researcher lacks control over how information is collected. 5. Units of Analysis and Variable Attributes a. A common problem in existing statistics is finding the appropriate unit of analysis. Many statistics are published for aggregates, not the individual. A related problem involves the categories of variable attributes used in existing documents or survey questions. This is not a problem if the initial data were gathered in many highly refined categories. The problem arises when the original data were collected in broad categories or ones that do not match the needs of the researcher. 6. Missing Data a. One problem that plagues researchers who use existing statistics and documents is that of missing data. Sometimes, the data were collected but have been lost. More frequently, the data were never collected. Issues of Inference and Theory Testing a. A researcher’s ability to infer causality or test a theory on the basis of nonreactive data is limited. It is difficult to use unobtrusive measures to establish temporal order and eliminate alternative explanations. In content analysis, a researcher cannot generalize from the content to its effect on those who read the text, but can only use the correlation logic of survey research to show an association among variables. Unlike the ease of survey research, a researcher does not ask respondents direct questions to measure variables, but relies on the information available in the text. 50