Developing Learning Objectives and Assessment Plans

Developing Learning Objectives and Assessment Plans at a Variety of Institutions: Examples and Case Studies More meaningful assessment plans are increasingly expected at colleges and universities in the United States. These plans are implemented with the intention of providing faculty with data that can be used to improve courses and programs. This attention to objectives, outcomes, and continuous improvement has emerged in the latest version of the guidelines from the American Chemical Society (ACS) Committee on Professional Training (CPT) (1). The new guidelines request a department's most recent self-evaluation and a plan for acting upon the recommendations (1). With both university- and discipline-based calls for data-driven course and curriculum improvement, it seems safe to assert that most chemistry faculty members will be involved in some form of enhanced assessment prerogatives for the foreseeable future. At ACS's 232nd National Meeting, eight faculty presented information on the progress toward more effective assessment practices within their program or institute (2-9).1 Table 1 provides a partial list of some of these departments and their assessment Web sites. As a method of disseminating current practices, this article reports assessment practices and trajectories of six chemistry (and biochemistry) departments, one chemical engineering department, and the ACS Examinations Institute represented in the symposium. Given the focus of the new ACS CPT guidelines on student outcomes, faculty who are not well acquainted with assessment practices may find examples and case studies useful. Taking the First Steps Assessment plans are generally formal responses to informal observations. In some cases, the motivation for an assessment plan comes from an internal observation within a department that student learning seems to be missing some important component. In other cases, the “observation” is made by an accrediting agency and is more global, rather than department focused. In the case of departments seeking ACS approval, the impetus comes from the new ACS CPT guidelines. Formal assessment plans spring from the clear articulation of what faculty members expect students to learn. Figure 1 illustrates how this first step shapes the development of learning objectives, the associated assessments, and data-driven decision making that can lead to improvements in a chemistry program. Thus, the first steps toward an effective, curriculumwide assessment plan are for faculty members, or a small group of faculty members, to establish goals and develop learning objectives. Establishing Goals and Priorities Chemistry departments might not take a common path toward establishing goals, priorities, and learning objectives. Purdue University's undergraduate committee spent 2.5 years reviewing the undergraduate majors' curriculum to establish priorities and to determine desired outcomes. To provide structure for the process, they used Wiggins and McTighe's Understanding by Design (10) to classify curricular outcomes as critical, important, or desirable. • Critical outcomes are considered to be vital and of fundamental importance. They are outcomes in which an enduring understanding is needed, such that students will remember them long after the details have faded. • Important outcomes are more specific and pertain to ideas or skills that the student must know or be able to do. Student learning is incomplete without mastery of these essentials. • Desirable outcomes are recognized as worth knowing, but the aim is exposure, not mastery. Each outcome was further classified by type as: • Technical competency: Operational skills • Technical competency: Knowledge based • Critical/Analytical: Thinking skills This classification system produced nine categories (three levels with three types) and 38 individual outcomes. For each course, an outcome was classified as: W, well addressed; Table 1. Institutional Affiliation and Departmental Assessment Web Sitesa Institution Web Site URLs Utah State University http://www.chem.usu.edu/assessment/ University of Wisconsin-Oshkosh http://www.uwosh.edu/departments/chemistry/assess/program.html University at Buffalo-SUNY http://www.cbe.buffalo.edu/undergrad/Improvement/ ACS Examinations Institute http://www4.uwm.edu/chemexams/ a This is a partial list from the “Development and Implementation of Learning Objectives in Chemistry Departments: A View of Progress at a Myriad of Institutions” symposium, which can be found in refs 2-9. Figure 1. Scheme showing iterative cycles of assessment. Table 2. Distribution of Outcomes for 100- and 200-Level Chemistry Coursesa a W = well addressed, S = somewhat addressed, or R = recommended to be addressed (shaded). S, somewhat addressed; or R, recommended to be addressed. This produced tables of outcomes, a portion of which is shown in Table 2 for 100- and 200-level chemistry courses at Purdue. Linking outcomes with each course facilitated the department's ability to analyze the data, and to develop recommendations. This methodology and time frame allowed for repeated engagement of all faculty in the process. However, some institutions need to develop an entire assessment plan more rapidly in response to an administrative mandate or an accrediting agency's report. Under these conditions, one of the challenges is to acknowledge the importance of faculty ownership and buy-in, rather than adopt a “response to mandate” approach that ignores this aspect of managing change. Developing Learning Objectives In the process of developing an assessment plan (see Figure 1), once priorities and goals have been established, learning objectives can be generated. Objectives are a statement of intended outcomes that can be measured. Well-written objectives are said to be “SMART”: specific, measurable, achievable, relevant, and timely (11, 12). These can be programmatic objectives such as the ones below from University of Wisconsin-Oshkosh (UW-Oshkosh) (13). Upon completion of a chemistry major, students will be knowledgeable about the factual and theoretical basis of chemistry. Specifically, the students should be able to: • Describe the structure and composition of matter • Plan the synthesis and characterization of inorganic and organic compounds • Apply theoretical and mechanistic principles to the study of chemical systems employing both qualitative and quantitative approaches • Use theories of microscopic properties to explain macroscopic behavior • Explain the role of energy in determining the structure and reactivity of molecules Course-level objectives can be more detailed. Utah State University (USU) has developed a matrix of course-level objectives Table 3. Bloom's Taxonomy of Cognitive Objectives with Example Verbs Objective Description Example Verbs Knowledge Terminology and specific facts Define, describe, identify, labels, match, state Comprehension Understanding Classify, convert, describe, explain, summarize, translate Application Use of learned information in new situation to problems (single answer) Calculate, construct, extend, produce, solve, transfer, use Analysis Deconstruct information to components develop conclusions by making inferences, find evidence to support conclusions Compare and contrast, correlate, differentiate, discriminate, illustrate, recognize Synthesis Creatively applying prior knowledge and skills to produce a new whole Adapt, categorize, collaborate, design, devise, incorporate, modify, negotiate, revise, validate Evaluation Judging the value of material Appraise, compare and contrast, criticize, critique, defend, interpret, justify, recommend, reframe accessible through their Web site (14). For example, in 100-level chemistry courses, the following objectives are measured: • Deduce chemical structures given chemical composition • Discuss and apply concepts of chemical structure and bonding to predict chemical structure and chemical reactivity • Compare and contrast the chemistry of metals, nonmetals, and semimetals • Be able to relate the microscopic and macroscopic properties of matter to each other • Use physical models to describe energies and forces in atoms and ions and explain the trends of the periodic table • Describe gas properties using molecular kinetic theory • Use the laws of thermodynamics to discuss and predict chemical reactivity and spontaneity Phrasing objectives is important if they are to be used as a foundation for assessment. Bloom's taxonomy of the cognitive domain has frequently been used as the guide for writing learning objectives (see Table 3), although Wiggins and McTighe's Understanding by Design is a good resource as well (10, 15). Bloom's taxonomy allows faculty to develop learning objectives that can be classified into a hierarchy. The action verbs in the taxonomy indicate the student behavior to be assessed. Beyond identifying verbs to use in writing objectives, some verbs should not be used. Verbs such as “know”, “understand”, “comprehend”, “grasp”, or “appreciate” should be avoided because they indicate behaviors that cannot be measured. Those verbs should be replaced with terms resulting in a measurable and specific objective that describes the behavior or skill to be assessed or demonstrated. For example, an objective in a biochemistry course could be written as “understand protein structure”. Using the word “understand” gives no indication what the student is expected to do to demonstrate an understanding. The objective could be rewritten as “given an image of a protein, describe the primary, secondary, tertiary, and/or quaternary structural features”. This objective explicitly defines the skill that will be assessed, the ability to describe protein structure based on an image. Designing and Conducting Assessments An assessment plan includes the design and collection of assessments. It provides evidence of how well or the extent to which students are meeting the objectives at either or both the course level and the programmatic level. Key facets of any assessment are ease of use, utility of the data, and leverage of current practices. Practices vary based upon the size of the student population being tracked, the focus of the evaluation or assessment (is it programmatic or at the student level?), and the institutional resources available to support the assessment. Four examples or case studies are detailed below. Example 1: University of Wisconsin-Oshkosh In 1995, the chemistry department at UW-Oshkosh implemented an assessment plan that encompassed the assessment of chemistry majors as well as students in general education courses (13). Chemistry majors compile a portfolio of artifacts from third-year-level and fourth-year-level courses, including final exam scores and ACS exam scores; graded laboratory reports; laboratory skills assessments; seminar papers and presentations; and grids linked to assessment of student performance with regard to specific departmental objectives. Rubrics developed by UW-Oshkosh faculty are used to classify student performance. The classification system for laboratory skills is shown in Table 4, and is known as an “NIA” assessment. Within the UW-Oshkosh assessment plan, each set of objectives includes a description of the evaluation process or instrument, evaluation criteria, standard of success, proposed program modifications if the standard of success is not achieved, and a timetable for implementation of the assessment sequence. Faculty members also combine grading with assessment procedures to streamline the collection of data. For example, student performance on clicker questions in general chemistry is Table 4. NIA Rubric for Laboratory Assessments Code Level of Skill Description N Novice Follows written protocols, performs under the explicit guidance of the instructor I Intermediate Follows written protocols, performs with minimal guidance A Advanced Interprets protocols and modifies to new conditions; designs protocols for others NA No expectation No expectation in this area linked to semester achievement on course objectives. Each clicker question is assigned to one of 22 course objectives, and then the database of student responses can be mined and analyzed to produce performance data linked to course objectives. Departmental assessment resources have also expanded by using administrative staff to track skill-level assessments (the NIA assessments) for each chemistry major (total of about 30-40 students). Example 2: Hope College Hope College began with a portfolio program much like UW-Oshkosh uses for their chemistry majors. Over time, faculty became inundated with artifacts (exams, quizzes, laboratory reports, etc.) to track and assess in a program that had over 120 chemistry majors (between 30 and 40 per year). Faculty reconsidered the assessment plan, moved away from the portfolio model, and developed a three-pronged data collection system that includes ACS exams, student self-assessments, and faculty self-assessment. The self-assessment tools were modeled after the SALG (16) and made use of its Web site to collect and analyze data. The use of ACS exams as finals allows faculty to compare student performance on nationally normed exams. In an institutional environment in which assessment resources are not in great abundance, the chemistry faculty found a way to leverage current practices;the use of ACS exams as finals;to their benefit. Example 3: Utah State University Utah State University's overall assessment plan has been developed and implemented at the course, subdiscipline, and programmatic levels. It uses a mixture of faculty-developed gain score tests administered the first and last week of the semester, ACS Exams, and student assessments as pieces of the plan. The nationally normed ACS general chemistry exams allow USU to compare the performance of their students with a national population. USU students compare well with the national norms, which has pleased faculty and added to the credibility of their program. USU has developed a capstone course for chemistry majors in which the students present a seminar on either their undergraduate research or an article from the research literature. In addition, faculty members have developed a multiple-choice capstone comprehensive exam for chemistry majors. Initially, faculty were surprised to find that only 37% of the students could determine a reaction order, and only 57% of the students could classify a titration curve. Faculty members have begun analyzing comprehensive student performance and are exploring ways to improve it. Example 4: University of Buffalo The University of Buffalo chemical and biological engineering department has developed software to gather and analyze data at the student, course, and programmatic level. Electronic exam or homework problems can be chosen at random and viewed for a random population of students in a specific course. From these data, conclusions can be drawn about the extent to which students are meeting objectives. Chemistry departments and engineering departments are governed by a different set of accreditation rules or guidelines. Engineering programs are accredited by ABET (Accreditation Board for Engineering and Technology) and are guided by the EC2000 criteria, which focus on assessing what students have learned, rather than what they have been taught. The EC2000 has helped engineering departments focus on continuous improvement of their programs guided by their individually crafted objectives and assessments. Thus, at the University of Buffalo, individual course objectives are aligned with outcomes for the engineering program. Evaluating Assessment Findings: Data-Driven Decision Making The results from the evaluation of assessment findings must be used to make data-driven decisions to complete the first cycle in Figure 1. For example, at USU the evaluation of assessment findings has led to changes at the course and programmatic level. One of the tangible outcomes has been the creation of a BS degree in biochemistry. USU has also increased the number of lecture hours in organic chemistry from three to four per week based on the analysis of student performance and feedback. In response to student concerns about receiving prompt feedback about their learning, the department uses electronic quizzes rather than hand-graded quizzes that often had a return time of over a week. The department is also using an electronic response system, I-Clicker (17), to increase the level of student engagement in large lecture courses. See ref 18 for further information about USU's experience. At UW-Oshkosh, faculty discovered that most of the students in the ACS-approved program were rated as “intermediate” or “advanced” on the programmatic goals that mattered most to the faculty. However, a more detailed analysis of the data led the faculty to make the inorganic and physical chemistry laboratories a required part of the lecture course rather than a stand-alone laboratory course. For the summative evaluation of educational objectives within the department of chemistry and biological engineering at the University of Buffalo, faculty included information from alumni surveys and conversations with employers and the dean's advisory council. From these data sources, the department has initiated efforts to improve students' technical communication skills, teamwork skills, laboratory safety skills, and experience with modern equipment in laboratory. The department has also implemented and supported faculty-based efforts to improve classroom practices and directly impact program outcomes. See ref 19 for additional information from the department's Web site. ACS Examinations Institute Role in Assessment The ACS Examinations Institute (EI) has been producing assessment materials for over 70 years. Committees of chemistry faculty and teachers intrinsically provide content validity to create the examinations. The examinations are trial tested and analyzed, which provides construct validity. Further, national norms for these exams are calculated and published online so that educators can compare the performance of their students and share the results with faculty and administrators. The ACS EI has initiated a process that will culminate in criterion-referenced exams. Tom Holme, current director of the ACS EI, has noted that the content coverage of the general chemistry exams has drifted relatively little in the past decade. Thus, a set of criteria can be built for the general chemistry exam. If such criteria could be generated across the curriculum, it would be possible to track knowledge growth for a student across four years, providing a meaningful route to programmatic assessment. The EI also plans to use cognitive load theory to build a system to establish the objective complexity of exam items (20). Thus, in addition to a measure of the content mastery of students, referring exam items to cognitive theories will also allow assessment of growth in cognition during the course of the undergraduate curriculum. The ACS EI has developed the Diagnostic of Undergraduate Chemistry Knowledge Examination (the DUCK), for use at the end of an undergraduate chemistry program. The committee of 20 faculty members who generated this exam was guided by the Exploring the Molecular Vision conference sponsored by ACS's Society Committee on Education (SOCED), which discussed a set of anchoring concepts that span the undergraduate chemistry curriculum (21). The DUCK is composed of 12 scenarios, each with four multiple-choice questions pertaining to basic knowledge, data interpretation, experimental design, and concept knowledge. Many of the colleges and universities use ACS EI examinations as part of their assessment plans. The exams offer departments the opportunity to compare the performance of classes of students to national averages through norms and to determine student content knowledge growth through criterion-referenced exams. The DUCK allows departments to engage in self-assessment via the cognitive skills demonstrated by the students who take this exam. Making use of this wellknown and credible resource is part of many departmental assessment plans. Iterating and Reiterating Cycles of Assessment Assessment is most useful when the cycle of events identified in Figure 1 is complete, the loop is closed, and the data is used to inform another cycle. For example, after the first cycle is complete, the assessment findings can be used to determine whether goals are being met, initiating a second assessment cycle. Changes can then be proposed to meet goals more effectively. These may include the modification of goals and priorities, and the revision of objectives. Assessments can be designed and conducted, and the assessment findings can then be evaluated and used to shape a third cycle. Thus, the iterative nature of the assessment process should lead to the continuous improvement of a program over time as noted in the ACS CPT guidelines (1). Conclusion Assessment plans are being formulated at many institutions across the United States. By sharing the development of these plans and relevant case studies through symposia, articles, workshops, and informal conversations, faculty can become aware of the work that has already taken place. No one needs to reinvent the wheel. In many cases, faculty can learn from and adapt plans that have developed and implemented at other institutions. In addition, resources from the ACS EI can help departments gather meaningful data through reliable and valid assessments to make data-driven decisions. Acknowledgment Thank you to Tom Holme of the ACS Examinations Institute; Joanne Stewart of Hope College; and Drannan Hamby for helpful comments on this manuscript. Note 1. Karen Muyskens, who contributed to the symposium noted in this article, passed away January 13, 2008 after a brief illness. She is deeply missed by her friends and colleagues. For further information please see http://www.calvin.edu/news/2007-08/ karen-muyskens.htm (accessed Sep 2009). Literature Cited 1. ACS CPT Guidelines: Undergraduate Professional Education in Chemistry. http://portal.acs.org:80/portal/fileFetch/C/WPCP_ 008491/pdf/WPCP_008491.pdf (accessed Sep 2009). 2. Farrelly, D. Learning Objectives and Assessment in Chemistry and Biochemistry at Utah State University. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, IL, March 2007. 3. Weaver, G. Examining a Chemistry Major's Curriculum from a Learning Objectives Viewpoint. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, IL, March 2007. 4. Kelly, R. Developing Learning Goals for a Chemistry Program. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, Il, March 2007. 5. Mihalick, J. E. Collecting Data on Student Learning and Using It To Improve a Chemistry Program. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, IL, March 2007. 6. Muyskens, K. C. Department Mission Statements and Assessment Plans: Lessons from the Trenches at an Undergraduate Liberal Arts College. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, IL. March 2007. 7. Stewart, J. Department Assessment of Learning Objectives at a Liberal Arts College. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, IL, March 2007. 8. Kofke, D. A.; Errington, J. R.; Schultzrogress, A. Progress in Developing a Web-Based Database-Driven Curriculum Assessment Tool. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, IL, March 2007. 9. Holme, T. A.; Murphy, K. ACS Exams and Programmatic Assessment. Presented at the 232nd National Meeting of the American Chemical Society, Chicago, IL, March 2007. 10. Wiggins, G.; McTighe, J. Understanding by Design; Prentice Hall: New York, 2001. 11. Drucker, P. F. The Practice of Management; Collins Business: New York, 2006. 12. Cozemius, A.; O'Neill, J. The Power of SMART Goals: Using Goals To Improve Student Learning; Solution Tree: Bloomington, IN, 2005. 13. UW-Oshkosh Department of Chemistry Assessment Plan. http://www.uwosh.edu/departments/chemistry/assess/program. html (accessed Sep 2009). 14. University of Utah Chemistry Department Undergraduate Program Assessment. http://www.chem.usu.edu/assessment/#courselevel (accessed Sep 2009). 15. Bloom, B. S. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain; David McKay Co., Inc.: New York, 1956. 16. Seymour, E. Student Assessment of Learning Gains. http://www. salgsite.org/ (accessed Sep 2009). 17. I-Clicker Home Page. http://www.iclicker.com/ (accessed Sep 2009). 18. Data-Based Decision Making. http://www.chem.usu.edu/assessment/databaseddescisions.php (accessed Sep 2009). 19. University of Buffalo Continuous Improvement of CE Undergraduate Program. http://www.cbe.buffalo.edu/undergrad/Improvement/index.php (accessed Sep 2009). 20. Knaus, K.; Murphy, K.; Picione, J.; Blecking, A.; Holme, T. A Valid and Reliable Instrument for Objective Complexity Analysis of Chemistry Exams. Unpublished work, 2008. 21. American Chemical Society. Exploring the Molecular Vision: Conference Report. http://portal.acs.org/portal/PublicWebSite/education/ policies/educationguidelines/CTP_005564 (accessed Sep 2009).