Web-based vs. Face to Face Learning of Diabetes Management - The Results of a Comparative Trial of Educational Methods

Web-based vs. Face to Face Learning of Diabetes Management: The Results of a Comparative Trial of Educational Methods Background and Objectives Relatively little is known about the effectiveness of web-based learning (WBL) in medical education, and how it compares to conventional methods. This study examined the impact of an interactive, online curriculum in a 3rd year medical school family medicine clerkship on students’ ability to create a management plan for a patient newly diagnosed with type 2 diabetes, and how the online curriculum compared to a conventionally taught method. Methods The online course included 3 integrated activities: 1) self-study modules; 2) a patient case-study; and 3) a moderated discussion board for posting and discussing patient care plans. The WBL curriculum was compared to smallgroup case-based sessions with a faculty facilitator. Students completed a test case pre and post clerkship. Results Among standard-of-care diabetic management interventions not ordered on the pretest, 38% were subsequently correctly ordered by WBL students on the posttest, vs. 33% by students in the comparison group (P<0.05). For 4 out of 5 subgroups assessed on the case write-ups, the gain from pre to post clerkship favored the WBL group. Conclusion Improvement among students learning online exceeded that of students learning in face-to-face groups. This suggests superiority of the online method, a finding consistent with other recently published, well-controlled studies. Introduction Courses in the clinical years of medical school often place students in preceptor arrangements in off-campus clinical sites with community-based faculty. Because of the many benefits of these educational settings, medical education experts have advocated acceleration of community-based teaching [1] Despite their benefits, students and faculty are often physically remote from the academic medical center [2] and have restricted access to conventionally structured curricular materials delivered at the medical school campus. Students in community-based clerkships may be placed alone and, relative to clerks on rotations at the academic health center, often have fewer educational interactions with peers or core faculty. Leaders in medical education have also challenged medical schools to “exploit the potential of distance learning technology to deliver educational programs in which instruction and evaluation are of a consistent and high standard across multiple settings in the community”. [3] The Association of American Medical Colleges’ Medical School Objectives Project has recommended the use of interactive Web-based courses to supplement and complement conventional curricula. [4] Properly designed WBL programs can serve to efficiently deliver educational programs to students, residents, or practicing physicians who may be far from each other and from medical school faculty. WBL can also help to standardize students’ learning, insuring that all students are exposed to essential curricular elements. However, given its recent introduction, relatively little is known about the effectiveness of WBL in medical education, and how this method compares to conventional face-to-face educational methods. This paper describes the design, teaching, and evaluation of a web-based curriculum for third-year medical students on a family medicine clerkship rotation. The content area selected for this WBL activity was type 2 diabetes mellitus, a disease of epidemic proportion in the US, and the second most common reason for visits to family physicians in the US.[5] The age-adjusted prevalence of diabetes has increased by 54% among U.S. adults during the period 1994-2002. [6-9] This study examined to what extent the implementation of an interactive, online curriculum in a family medicine clerkship improved students’ ability to create a management plan for a patient newly diagnosed with T2DM, and how the online curriculum compared to a conventionally taught method. Methods Design and Content of the Web-based Curriculum Our approach to WBL course design has been described previously. [10-12] It is guided by educational psychology principles including adult learning, reflective learning, and collaborative learning. [13-15] The content of the WBL curriculum was adapted from an existing face-toface (F2F) case-based curriculum, originally created according to a simulated family model, [16] designed to promote competence in the diagnosis, initial evaluation, and management of a patient newly diagnosed with T2DM. The online course included 3 integrated educational activities: 1) self-study modules; 2) a patient case-study; and 3) a moderated discussion board for posting and discussing patient care plans. Students first studied an online module, composed of 12 html pages with text and interactive features, addressing the diagnosis and initial management of T2DM diabetes based on current guidelines. [17] Next, they applied these concepts to a case study presented in 3 forms: a streaming video of the case patient describing her symptoms; a simulated electronic medical record; and a brief written case summary. The case patient was a middle-aged woman with symptoms of DM and risk factors for T2DM including obesity, Hispanic ethnicity, family history of T2DM, and hypertension. After reviewing these materials, the learner then described her assessment and management plan in a posting to an online asynchronous discussion board. The discussion board activity represented a reflective learning step in the curricular cycle. These discussions were moderated by a trained faculty member who provided feedback, asked for clarification, and probed the learner for deeper reflection and understanding using established principles of moderating online educational discussion boards. [18] To encourage peer learning, all students in the discussion groups were required to read and comment on at least one other student’s posting each week. With each passing week, the online diabetes case reflected passage of time and progression of disease with new management challenges. In addition to studying type 2 DM online, clerkship students also completed a web-based EBM curriculum and an online medical humanism journal activity. Blackboard [19] was used to organize course materials and activities for all 3 curricula, collectively called the Online Clerkship (OC). Students were required to access the OC at least twice weekly and were provided protected time to do so. The Comparison Face-to-Face Curriculum The WBL curriculum was compared to our standard clerkship didactic educational activity: small-group case-based sessions with a faculty facilitator. These sessions occurred on three days when students were not seeing patients. They were provided with outpatient medical charts for a simulated family [16], and discussed management plans for each visit of member of the simulated family. This curriculum addressed a variety of common outpatient medical conditions (such as T2DM, low back pain, asthma, pregnancy, urinary tract infection, alcoholism, and others) over simulated passage of time. Students who were in F2F groups were taught the principles of management of type 2 DM using this seminar approach, and had no online assignments. Like students in the WBL groups, the assignment for students in the F2F groups was to discuss and develop a management plan for a patient newly diagnosed with type 2 diabetes. Students had access to printed diabetes care guidelines comparable in content to material accessed by online students in their html-based module. Each weekly diabetes case encounter, both WBL and F2F version, was designed to require equivalent duration of student engagement (about 1 ¼ hour). Small groups were facilitated by 4 department faculty who, in order to promote standardization of curriculum presentation and teaching process, underwent formal training in the curriculum in faculty development sessions, and utilized a written guide for faculty moderators, The Study Sample and Evaluation Methods Data were gathered from third-year medical students participating in the Boston University School of Medicine family medicine clerkship from January, 2001 through June, 2002. Alternating blocks of clerks were assigned to the WBL group (blocks 2,4,6,8). Control group students in odd blocks (1,3,5,7) participated in the F2F curriculum. Clerks completed surveys that recorded baseline demographic data, and their experience with the WBL curriculum. To evaluate the impact of the DM curricula, at the start and end of the clerkship on the final testing day, students wrote an assessment and management plan based on a paper-based case of a middle-aged male patient who presented to the office with lab values sufficient to diagnose T2DM. The validity of written case-based approaches to clinical skills assessment has been established. [20] The student management plans were completely blinded by a research assistant as to student name, date, clerkship block, pre or post status, and intervention group status. Each management plan was rated by one of the authors (JW) using a checklist based on then-current clinical guidelines from the American Diabetes Association for the content of the initial visit for a newly diagnosed patient with type 2 DM. [17] The case was constructed such that drug therapy was unequivocally indicated. The general domains rated in the 32-item checklist included: appropriate additional history and review of systems sought; appropriate physical exam planned; assessment (diagnosis of type 2 DM stated); plan (correct labs ordered and appropriate drug therapy recommended); and referrals and counseling documented. The same case was administered pre and post clerkship. Students were aware that scores on this assessment were not included in their final clerkship grade determination. In addition to comparing overall pre and post scores, a subset analysis was done of those interventions not ordered on the pretest to measure what proportion were subsequently correctly ordered on the post test. The effectiveness of the intervention was tested by using the Mantel-Haenzel procedure. This statistic was computed using the mantelhaen.test function in SPlus- version 7. [21] The study was approved by the Boston University Medical Center institutional review board. Results The final evaluation case was completed by 159 (88.3%) of clerkship students, with a similar response rate in both study groups. There were no significant differences between students in the online versus comparison groups in terms of age, gender, race/ethnicity, MCAT scores, preclinical grades, or previous Internet use patterns. Students reported spending an average of 4.3 hours weekly on all 3 activities in the online clerkship. Students accessed the project website from home (61%), preceptor office (24%), medical school (10%), and the remainder from other locations. Results from the evaluation cases are shown in Table 1. Students in the WBL group showed a statistically higher post-test score, and a greater gain in overall score from pre to post-test, than did students in the non-online group. Subgroup analysis shows that for 4 out of 5 categories assessed on the case write-up, the difference in gains favored the WBL group, although the difference in gain for only one subgroup (“Plan”) reached statistical significance due to diminished statistical power resulting from small subgroups. Among subjects who did not order an ADA-recommend intervention at baseline, subjects in the WBL group were more likely than students in the F2F groups to correctly order the intervention on the post -test (Table 2). Students reported roughly equivalent attitudes towards enjoyment of the 2 curriculum formats, and indicated similar impressions of effectiveness (Table 3). Most students felt the WBL curriculum was easy to use, and over 50% reported that the online discussion groups were valuable (Table 3). Discussion This study demonstrated that improvements in student performance on a casestudy among students learning online exceeded that of students learning in faceto-face groups. This suggests at least equivalence, if not superiority, of the online method, a finding consistent with other recently published, well-controlled studies evaluating effectiveness of online learning in medical education. [22] [23, 24] There was no significance differences in student attitudes with respect to effectiveness or enjoyment of the two learning methods, suggesting that both may be equally acceptable from a learner satisfaction standpoint. In the online group, students were able to learn from engagement in a curriculum that included no face-to-face instruction with on-campus faculty. In both experimental and control groups, the overall post-test results were only modestly superior to baseline performance. Three possible biases may account for this finding and illustrate the complexity of carrying out research in real educational settings: first, the pre and post-tests were not used in determining the final grades of students, which may have contributed to suboptimal student effort, whereas the other tests administered on the last clerkship day were used to evaluate and grade student performance. Second, the content in the casestudy was not included in the student course learning objectives, so that students would be less likely to study the material using methods other than the face to face or online groups. This was an attempt to isolate the learning associated with these methods only. Third, we suspect that students may have exerted less effort on the post-test versus the pre-test. The post-test environment, following a day of other testing, may have reduced student effort and biased scores downward, reducing the apparent effectiveness of both curricula. The pre-test was administered in the morning of the first clerkship day while students were fresh, whereas the only opportunity for administration of the posttest was at the end of a full day of testing. The pretest was the only survey given on day 1, whereas the post-test was the last of 4 tests. Time of day and sequence of examinations has been demonstrated to cause statistically significant variation in student performance on examinations. [25] It seems possible that given these factors, testing fatigue between the pre and posttests might have occurred. Nonetheless, such fatigue would have been experienced presumably equally between the WBL and control group students, and does not invalidate the observed favorable results of the WBL group. The subgroup analysis of gain, among those interventions not ordered at baseline, was included to demonstrate the relative impact of the 2 teaching methods, while isolating to the extent possible the influence of differences in the pre and post testing environments. This subgroup analysis showed that students in both groups improved substantially, and those who learned online were significantly more likely than controls to order interventions on the post-test that they had neglected to order at baseline. Of note, the most significant difference between performance of WBL and F2F groups was in the ability to suggest an appropriate treatment plan, including lab screening and pharmacotherapy. The substantial superiority of the online method of learning in these key domains might be attributable to the requirement in the online curriculum that students write, and post for feedback, a management plan for the case patient, whereas in the F2F group oral discussion is used to develop consensus about a management plan, which is written on a black board by one student. In the F2F groups, it is possible for students to participate and engage in the discussion and planning at more variable levels than is possible online. In other words, students can be relatively disengaged during F2F sessions, whereas this is less possible among Web-learners given the design of the online sessions. Furthermore, the process of writing and submitting for peer and faculty review is likely to encourage deeper reflection about the case, and supporting material. Learning styles differ from student to student [26] [27]. These differences incorporate student preferences as well as ability to learn, integrate and utilize data from different sources. It would be surprising therefore if an educational initiative found the best way to teach a group of students. A rich environment with a variety of teaching and learning techniques and options for students to choose amongst those techniques is likely to be a more profitable approach, and WBL is proving to be an important new option. Future research needs to define how to map WBL design to cognitive and learning styles [28] This randomized controlled trial of a specific teaching and learning method, asynchronous web based instruction, demonstrated the ability of students to learn diabetes management skills as well as a face to face seminar. This study has two important implications: students can be taught effectively using asynchronous communication on the WWW, and, web-based curricula improve the geographic availability of teaching and paves the way for further decentralization of medical school instruction. The findings on the effectiveness of this WBL education approach to EBM training also add to a growing and generally favorable body of literature on online learning. [23] Although “evaluation of web-based learning (WBL) is in its infancy” [29], the comparative research on distance-learning technologies is massive, dating back to the early 20th century. [30-32] A recent review noted that overall WBL appears to be equivalent in knowledge gains to other educational methods. [29] We have subsequently used the methods developed in this research in a number of other projects, including international CME training and patient education [11, 33] Leaders in medical education have called for medical schools to rely more heavily on high quality distance learning [3]. In response, we have developed a new WBL method of teaching key components of diabetes care. With these additional favorable data on the effectiveness of WBL, we anticipate that medical educators will increasingly take advantage of emerging distance learning technologies to improve the quality of undergraduate, graduate, and postgraduate medical education.