Using ExamSoft to Improve Major and Program Learning Outcomes in Biology

Using ExamSoft to Improve Major and Program Learning Outcomes in Biology Biology Department by the numbers • Approximately 900 major students • Average 225 per graduating class • 51 faculty engaged in teaching • 69 open-enrollment courses dedicated to our majors Biology Major Learning Outcomes content • Demonstrate knowledge of fundamental principles spanning the breadth of biology. Cognitive • Demonstrate expertise in the scientific method, including skills experimental design, critical assessment of the scientific literature, and an understanding of the principles and best practices for the ethical conduct of research. • Attain the technical and/or analytical skills required for employment or post-graduate education in biology or biology-related careers, including professional careers and science education. Lab skills Current means of assessment • Incentivized, optional senior survey • 3 questions on cognitive skills (data analysis & scientific method) • 11 questions spanning 4 biology content areas • 8 questions of self-assessed competency in cognitive and lab skills Percent Correct by Content Area 100% 90% 80% 70% 60% 50% 40% 30% 20% information flow systems (ecology) (genetics) structure and sci method/data function (cognitive skills) (macromolecules) Challenges • Limitations of data • Tiny portion of the breadth of biology assessed • Small details within • Low response rate (35%) • Low question number, may bias data • Creation and analysis takes time • Self-reporting can be flawed • Must keep survey short to encourage participation ExamSoft Software Online exam-taking software that delivers exams electronically and prevents cheating by “locking down” and securing students’ computers from other programs. Questions are categorized using Bloom’s taxonomy, textbook chapter, or program assessment outcome Student Feedback • Rapidly generates customizable personalized reports for each student • especially useful for large classes • detailed feedback and identifies areas of improvement • Data-driven conversation about learning strategies Multi-Year Analysis • The students and faculty can track performance over multiple classes and from one semester to another. • 2017: funding to test ExamSoft • program assessment • enhance metacognition in our students ExamSoft Program Assessment Data 100 40 items 2044 responses 15 items 1860 responses membrane transport data analysis 50 items 2044 responses 11 items 2044 responses 80 70 3 items 79 responses 60 50 40 1 item 79 responses % Correct in Content Categories 90 30 20 10 0 100-level 200-level macromolecules 300-level enzymes Senior survey • Seniors are under performing on survey (or forgetting?) • We can see where we need to increase skillbased content • Data are far more robust • Easy to target areas of missing data Use by Bloom's Level Across the Department Lower-level courses focus on remembering and understanding, while providing practice on applying and analyzing 85 At the 300-level, assessment is equally spread among higher and lower-order cognitive skills 80 100-level: 2 classes, 7 faculty 200-level: 1 class, 1 faculty 300-level: 1 class, 2 faculty Remembering % Correct Understanding 75 Applying Analyzing Evaluating 70 A new skill is introduced and assessed 65 60 100-level classes 200-level classes 300-level classes Metacognition: awareness and understanding of one’s own thought processes • Procedural knowledge: There are different study strategies that can be used • Conditional knowledge: Knowledge about when and why to use learning strategies • Information management strategies: Ability to organize, summarize, compartmentalize information learned • Comprehension monitoring: What are learning strategies, are you progressing in your learning? • Evaluation: Did the strategies used help you succeed at a task (exam)? Metacognition Study • Recent research indicates that metacognitively aware learners are more strategic and perform better than unaware learners • plan, sequence, and monitor learning in a way that directly improves performance • Our study • Pre- and Post-course survey after 1 semester with ExamSoft • Used a single course without ExamSoft as a control • Metacognitive Awareness Survey: 52 True/false statements such as • I am aware of what strategies I use when I study. • I know when each strategy I use will be most effective. • I ask myself how well I accomplish my goals once I’m finished. ExamSoft Significantly Accelerates Metacognitive Gains 0.45 ExamSoft helps students understand 0.4 what learning strategies are Increase (post-pre) 0.35 All students learn which strategies help the during exams, but students have more improvement with access to ExamSoft * * 0.3 Access to ExamSoft helps students gain information organizing skills 0.25 0.2 0.15 * * 0.1 0.05 0 -0.05 Procedural Knowledge Comprehension Monitoring courses with ExamSoft Evaluation Conditional Knowledge course without ExamSoft Information Management Strategies * Denotes statistical significance between pre and post scores within the category Is there are “right time” for metacognitive gains? 0.5 Students gain the most from ExamSoft earlier in their education 0.4 0.3 0.2 Increase (post - pre) Gains in comprehension monitoring did not show differences across class year 0.1 0 -0.1 freshman sophomore junior -0.2 -0.3 n 176 175 86 32 -0.4 freshman sophomore junior senior -0.5 Evaluation Information Management Strategies senior Current and future directions • Refine program assessment learning outcomes based on these data and data from faculty survey on classroom practices • Begin data-driven conversations among departmental faculty • Further statistical and longitudinal data analysis—data collection continues through ’18-’19 academic year • Development of Hub categories within ExamSoft • Cost/benefit analysis and sustainability Student Attitudes toward ExamSoft • Students were taught about Bloom’s taxonomy • Presented with analyses of their exam performance after every exam • Class exam data was presented and reflected on in lecture after every exam • Students were given study method suggestions from published educational literature for improving their learning in Bloom’s categories Do you agree or disagree with this statement: Examination of my "strengths and opportunities report" helped me to refine my study habits/techniques? strongly disagree 1.70% disagree 9.50% strongly agree 9.50% somewhat disagree 6% agree 23.30% neither agree nor disagree 14.70% somewhat agree 34.50% Do you agree or disagree with this statement: Over this semester or semesters of taking exams using examsoft, I have learned about myself as a learner. disagree 6.90% strongly disagree 1.70% strongly agree 12.00% somewhat disagree 6.80% neither agree nor disagree 14.50% agree 32.50% somewhat agree 28.20% Did Examsoft contribute to your understanding of different question types (i.e. Remembering vs Application). defintely not 3% probably no 19% definitely yes 18% might or might not 18% probably yes 41.8% Did you change your study habits over the course of this semester? no changes 17% yes, only small changes 53.20% might or might not have changed 5% yes, a lot of change 24.80% Do you prefer taking exams by examsoft or on paper & scantron form? paper and scantron 25.40% examsoft 56.80% no preference 17.80%