Evaluating Simulation Experiments in Chemical Engineering Courses

EVALUATION OF COMPUTER SIMULATION EXPERIMENTS IN A SENIOR LEVEL CAPSTONE CHEMICAL ENGINEERING COURSE ABSTRACT A systematic evaluation was done on the computer simulations developed for use in the capstone laboratory course in Chemical Engineering at Purdue University. The study suggests that computer simulations fill a different role than traditional experiments; that the environment in which the simulations are implemented has a major effect on students’ perceptions of their value; and that qualitative interview data can play a vital role in understanding the effect of changes in curriculum. INTRODUCTION In 1986, the School of Chemical Engineering at Purdue University began the revision of its senior-level capstone laboratory courses, which has included the development of a series of computer simulation experiments described elsewhere [1-7]. For each computer simulation, the students are given a budget (i.e., $35,000) that is the amount they can spend on experimental runs, wages, and consultation fees. The computer also keeps track of the “virtual” time the students used for each run and charged extra for work that would have to be done on weekends. This paper describes the results of an evaluation of the effect of using these simulations. This work was based on three assumptions. C C C When we change what we teach, or how we teach, we change what students learn. A systematic evaluation should be done whenever major changes are made in an established curriculum. Systematic evaluations should look behind the facade of answers to the question, “Do the students like it?”, toward deeper questions, such as “What do students learn that they were not learning before?” and, “If we could provide students with a voice, to express their opinions and concerns, what changes would they recommend?” The basic research question behind this study was: “How do the students’ experiences with the computer simulations compare with their experiences with traditional laboratory experiments?” Corollary research questions included: “What did the students perceive to be a valuable experience in both laboratory formats?” “How did the students’ decision making processes and other group-related interactions differ between the two formats?” And, “What do the students believe makes the computer simulation experiment a legitimate exercise to include in the Chemical Engineering curriculum?” -1- DEVELOPMENT OF EVALUATION METHODS This study was based on a collaboration between members of a Chemical Education research group [8] and faculty and staff from the School of Chemical Engineering who had developed and implemented the computer simulations. We began by scrutinizing a list of questions generated by Professor R. G. Squires and Dr. S. Jayakumar for use in a quantitative study of student attitudes toward the simulations. Some of the questions were retained; others were modified to make them either less complex or less “leading.” The result of this review was a 24-item five-point Likert-scale questionnaire that included space for students to write additional comments and/or suggestions. The questionnaire was given to the students after they had completed both a traditional experiment and a computer simulation experiment. Results of this survey for students from two semesters are summarized in Table 1. The authors developed a qualitative component of the evaluation [9] based on structured interviews with individual students or with groups of students; observations and field notes collected in the laboratories; written comments from the surveys described above; and interactions with the students in the labs. As those familiar with qualitative techniques might expect, the qualitative component provided the “richest” source of data for this study. The collection of qualitative data began with the researcher sitting in a corner of the traditional lab, taking field notes as he observed what was happening. The students would frequently start up conversations with the researcher about what he was doing there, what they thought about the experiment they were doing, and what they thought or had heard about the computer simulation experiments. Frequently the students would physically point out things that were working or not working with their traditional experiments, which helped the researcher gain some understanding of the experiments the students were performing. As these interactions continued, the researcher found it useful to switch from the role of an objective observer, sitting in a corner of the room taking notes, to that of a participant observer, listening to and talking with students while they worked. The students also seemed more com-fortable with this approach. The result was an environment in which a good rapport was developed between the researcher and the students before the structured interviews were conducted. This approach also provided the researcher with a set of experiences obtained while working with the students in the traditional lab that allowed him to prod the memories of the students during the interviews when they were asked to compare the two different laboratory formats. Observations collected while students were working on the computer lab did not prove useful because most of the decision-making process had already been accomplished during group meetings before the students came to the lab and the students were less likely to involve the researcher in their activities while they worked with the computer. Insight into these group meetings and the interactions between members of the group was provided by the structured interviews, however. -2- The structured interviews were the core of the qualitative evaluation methods. The researcher developed a list of questions that he wished to cover during the interviews, which covered many of the same topics as the Likert scale surveys so that the researcher could triangulate his conclusions from different data sources [10]. The use of the structured topic list produced interviews that followed a similar pattern, but the students had ample opportunity to bring up any subject they felt appropriate. The interviews were recorded, transcribed, and then analyzed using the method of inductive analysis [11]. The analysis consisted of reading the transcripts multiple times and condensing the student’s comments to common and uncommon categories by literally cutting and pasting together similar comments obtained in different interviews. QUANTITATIVE RESULTS AND DISCUSSION The results of the Likert-scale survey indicate that the students liked using the simulations (91%, Q 1); found the simulations easy to learn and operate (95%, Q 6); reported that the computer simulations did not intimidate them (86%, Q 7); would like to see more of them (73%, Q12); believed that the computer simulations allowed them to study more complex and realistic problems (78%, Q 18); valued the budgetary constraints included with the simulations, which made the simulations more realistic (86%, Q 3); and believed that they spent a higher percentage of their time planning the design of the computer simulation (86%, 17), which suggests that the simulations provide the students with an experience that is different from the traditional lab. The students liked the simulations for a variety of reasons, including the fact that they were more likely to work than the traditional experiments (81%, Q 23), thus giving the students reasonable and workable data. The computer simulations were very different from traditional labs because of the speed with which data could be acquired. This did not bother the students or make them feel uneasy about the computer experiment (89%, Q 8). In fact, they felt that this made more efficient use of their time (95%, Q 19). The students felt the simulations allowed them to focus on the principles involved in an experiment (82%, Q 9) and therefore were a good way to learn new processes and concepts (79%, Q 21). However, a majority (54%, Q 14) of the students felt that the traditional experiments gave them a better sense of the problems likely to be encountered in industry. Thus, it is not surprising that a majority (57%, Q 24) felt that the present combination of computer and simulation experiments was appropriate. The quantitative results produced a sense of conflict, or dualism, in the students’ opinions. They simultaneously believed the computer simulations are a good instructional technique, which helped them to better focus on the principles they were expected to apply, and yet the traditional experiments gave them a better sense of the problems they might encounter in industry. The source of this dualism cannot be extracted from the results of a Likertscale survey, but they can be obtained by triangulating this data source with the results of qualitative research techniques. As we will see, the students simultaneously regarded the computer simulations as both “good” and “bad.” They are good because they allowed the -3- students to tackle more complex problems, in which they were compelled to proceed with realistic budgetary and time constraints, and because these experiments were more likely “to work,” providing the students with data that allowed them to complete a realistic scaleup. The simulations are bad because they are not real; they cannot fail the same way a traditional experiment would fail. Even though the students tended to value the ability to focus upon important conceptual engineering issues in the simulation experiments, they recognized that this “ability” has little to do with the world in which they actually live. QUALITATIVE RESULTS AND DISCUSSION Twelve students were interviewed after they had completed one experiment of each type. The theoretical framework for this portion of the study falls within the domain of hermeneutics, [12] in the sense that we are trying to give students the opportunity to be heard, to have a “voice,” through interpretations of the meanings of their statements and actions. The interviews were used to probe more deeply into the students’ experiences, opinions, and beliefs about traditional versus computer-simulation experiments; to probe how students constructed the knowledge they gained from doing the lab experiments; to examine how they perceived computer simulation experiments, e.g., as just one long equation to be worked out with data generated by the computer, or as a chance to do meaningful engineering work similar to that done in modern industry; to explore their opinions on whether the computer simulations were more realistic or less realistic than traditional experiments; to discern whether the simulations require a particular teaching style from the instructor; and to determine the aspects of the computer simulation that make it more (or less) difficult than the traditional experiments. In some ways, the students felt the computer simulation was more realistic, and perhaps more difficult, than their other experiments.1 I: You were talking about the computer simulation being more “in-depth.” What did you mean by that? Andy: Instead of dealing with the unit, you dealt with the more of what you'd deal with in the real plant. ... the computer interfaced you to multiple types of equipment and more "real" equipment that you would use in industry rather than just the small glass tube that we used for the cation exchange. And I thought that was better because you get more of a full view of the operation rather than just one small aspect of it. Jody: In addition to that too we had a budget that we had to follow. Which is gonna be true in real life once we graduate and do what we need to do to get data and stuff like that. 1 In the narrative vignettes taken from the interviews, “I” stands for the interviewer and the names are nicknames given to protect the students’ identities. -4- The time and budgetary constraints imposed on the computer simulation experiments had the tendency to change the students’ decision-making process by forcing them to reflect on their decisions before taking action, as illustrated by comments made by Adam and Don, who were in separate groups. Adam: It made it more “real-world” I guess. Before on the other experiments if you wanted to ask the professor a question, we’d just go up and ask; even if it was just a stupid question. Now if we wanted to talk to the professor it would cost us $500 for a consulting fee. It made you stop and think about it instead of just running up and asking the professor when you could have figured it out yourself if you’d just have thought about it. Don: It was good to have a budget. If there was no actual planning involved, with no budget, we would just have run it for hours and hours and had stacks paper for results. We wouldn’t have thought about what we were doing. These comments are echoed by the results of the survey, which showed that the majority of the students felt that the use of the budget and time constraints made the simulation more realistic than the traditional laboratory experiments. Darrin and Laura found the realism introduced by the budget/ time constraints intimidating. I: Let me ask you about the computer simulation. What did you think of it when you first saw it? Darrin: Heh! Intimidating. I: How? Darrin: Well, even though we sat through a whole lecture I felt that I really didn’t know where to begin, and really I was ready to get another apparatus - experimental problem. With this [comp. simulation] I had no idea how to start. I was afraid that I was going to make a mistake. ... And plus there’s this thing that if you ask a question it would cost you like $500 or something. [consultation fee] So you’re kinda tentative. Laura provided insight into why her group felt intimidated by the computer experiment when she responded to a question that asked for her impression of the computer simulation. Laura: I was scared because it wasn’t like any of our other labs were, even if you, like, totally get bad data ... you don’t have anything to lose. You can still write up your report and say that your results are no good. But on this lab [computer simulation] you have to find your constants. -5- The problem was simple — there was no place for the students to “hide.” They could not gloss-over or “fudge” poor data collected during the computer experiment, the way they said they could when discussing traditional experiments. Darrin and Laura’s comments are not representative of the perceptions of the group of students who completed the computer experiment during the evaluation, but their comments raise an important issue in evaluation. Historically, evaluations of curriculum reform projects have been based on what we have called a “sports-mentality” approach [13]. Statistical techniques, such as a t-test on the mean scores of some measure of performance of students in experimental versus control sections of the course, are used to answer the question: “Is the new curriculum better or worse than the old curriculum?” Darrin and Laura’s comments remind us that any substantive change in curriculum will have both positive and negative effects. Some students will benefit, others will be hurt. Evaluative studies, such as this, allow one to search for both effects and then probe what additional changes could be made to maximize the positive effect and minimize the negative effect. Darrin and Laura’s interviews identified another source of differences in students’ perceptions of the computer simulations — the amount of success the students felt they had enjoyed. In general, the students who were interviewed felt that they had enjoyed success with the computer simulation. However, Darrin and Laura’s group did not share this perspective, as illustrated by the following comments. I: What was the computer simulation supposed to do and what did it really do as you look back on it now? What was it supposed to represent? Darrin: I think that it was supposed to represent a better way ... of solving a large problem that we could never have solved on a laboratory scale. With the amount of trials we ran .... it was supposed to demonstrate how much work you could get done, ... how many trials you could get done on the computer. But what it turned out to be was just trial and error. Laura: I think that what the ... simulation was doing was to show us how we can use a computer to simulate something and then to optimize conditions. And then apply them to an actual plant or whatever. And, ... I guess it did it, .... I don’t know! I don’t really know because I still don’t really understand how our values correlated to the actual running of the simulation and the running of pilot plant. ... I don’t really think that I learned anything from it. I just learned to manipulate what we were trying to do ... I’m still a little unclear on some things. The interviews provided useful information about the students’ perception of the role of the computer in their response to the question: “Which experiment gave you the best experience?” -6- Dallas: [The Computer Experiment] was the best. Granted, that they wanted to try to give us some sort of real life simulation ... what’s its like in real industry. ... We were actually able to get our numbers and do our scale-ups, and do our actual engineering work without trying to mess around trying to get something to work. Or trying to get some data or making up data ... we were actually able to do engineering. Don: [In the computer experiment] you could kind-of estimate sort-of where ... what would happen under certain circumstances. Whereas in the experimental part of the regular lab, you would get such confusing results. It was so difficult to try to extrapolate that onto any large scale. We basically said, “We’ll just have to throw this out and we’ll see what somebody else did or make up something.” These comments reflect the common perception among the students interviewed that the simulations were more likely to work than the traditional experiments; that they could acquire feasible data from the computer simulation. And yet other students questioned whether the computer simulation gave them the best experience. Adam: The best experience was probably with the water cation exchange, just because we did a lot more research with that to learn how to get the right data and stuff like that. The computer simulation was interesting but pretty much all the data was right there in front of you. ... The computer simulation was pretty neat. But it was a lot of wasted time for three people to sit there and do it, because only one person could get on the computer and run it. Ruth and Tina provided another insight into the computer experiment. Ruth: With the computer experiment all we did was calculations, ..... with the spreadsheet and stuff. That’s all we did. The whole lab was one big long calculation. Tina: The whole lab was just finding numbers you had to put in the simulation. You had to work through a bunch of equations. THE EFFECT OF THE PROFESSOR’S TEACHING STYLE Unlike surveys, which can only provide answers to questions that are explicitly stated, interviews often provide data on topics or questions one might not have anticipated. Consider the role of the professor’s teaching style, for example. This topic was not covered in the survey, but the interviews showed that it had a significant impact on the students’ experiences. It was clear from the interview data that the Professor’s “hands-on” teaching style during the planning sessions had a direct impact on the students’ perception of the computer simulations and the success of these simulations. -7- CONCLUSIONS The results of this study suggest that it would be a mistake to ask which laboratory format is “better” for students. They indicate that computer simulations and traditional experiments have different roles in the curriculum because they emphasize different aspects of engineering and require both different levels and types of expertise. Students who were frustrated with traditional lab equipment seemed to enjoy “actually doing the engineering” required to tackle the complex problems provided by the computer simulation. They did not have to worry about “making up data ... or seeing what someone else did” when the traditional lab failed. For these students, the computer simulations were more “realistic” than the traditional lab that gave results students described as “... I’d turn it in for a grade but I certainly wouldn’t buy it!” For other students, the simulations were less “realistic” because they cannot fail, the way a traditional lab fails. This study provided insights into the role of the environment in which computer simulations are implemented. Our results clearly indicated that the budgetary and time constraints played an important role in making the computer simulations seem “realistic,” so realistic that a few students felt intimidated by this aspect of the simulations. This study also suggests that computer simulations, by themselves, are not magic bullets that provide instruction and pedagogical benefits for the students in the absence of a human interaction between the students and the instructor. They are best thought of in terms of being a tool for instruction, rather than a replacement for the instructor. The authors hope that this study leads others to recognize the importance of asking the correct questions when evaluating curriculum reform projects, as required by ABET and NSF, and the importance of collecting qualitative interview data to both reinforce quantitative data collected in anonymous surveys and to provide a deeper understanding of the effect of curriculum changes on students’ attitudes and opinions. ACKNOWLEDGMENT The authors wish to thank Professor Robert Squires and Dr. S. Jayakumar of the School of Chemical Engineering for their support throughout this evaluation study. REFERENCES 1. 2. Squires, R. G., Reklaitis, G. V., Yeh, N. C., Mosby, J. F., Karimi, I. A., and Anderson, P. K. Purdue-Industry computer simulation modules: The Amoco resid hydrotreater process. Chemical Engineering Education, 1991, 25(2), 98-101. Squires, R. G., Anderson, P. K., Reklaitis, G. V., Jayakumar, S., and Carmichael, D. S. Multimedia-based educational applications of computer simulations of chemical engineering processes. Computer Applications in Engineering Education, -8- 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 1992, 1(1), 25-32. Jayakumar, S., Squires, R. G., Reklaitis, G. V., Anderson, P. K., Graziani, K. R., and Choi, B. C. The use of computer simulations in engineering capstone courses: A chemical engineering example — The Mobil catalytic reforming process simulation. International Journal of Engineering Education, 1993, 19(3), 243-250. Jayakumar, S., Squires, R. G., Reklaitis, G. V., Anderson, P. K., and Partin, L. R. Purdue-Industry computer simulation modules 2. The Eastman Chemical reactive distillation process. Chemical Engineering Education, 1993, 27(2), 136-139. Jayakumar, S., Squires, R. G., Reklaitis, G. V., Anderson, P. K., and Dietrich, B. K. The Purdue-Dow styrene-butadiene polymerization simulation. Journal of Engineering Education, 1995, 84(3), 271-277. Jayakumar, S., Squires, R. G., Reklaitis, G. V., and Grassi, K. S. Simulating the Air Products cryogenic hydrogen reactive cooling process. Chemical Engineering Education, 1995, 29(1), 26-31. Squires, R. G., Kuriyan, K., Jayakumar, S., Reklaitis, G. V., Evans, M., Morrato, B., and Gutwein, R. (1996). The Procter and Gamble decaffeination project: A multimedia instruction module. Computer Applications in Engineering Education, 1996, 4(4), 269-274. Bodner, G. M., Herron, J. D. Completing the Program With a Division of Chemical Education, Journal of College Science Teaching, 1984, 14(3), 179-180. Patton, M. Q. Qualitative evaluation and research methods (2nd. ed.). Newbury Park: Sage (1990). Ibid. pp 187-189. Goetz, J.P., and LeCompte, M.D. Ethnography and Qualitative Design in Educational Research. San Diego: Academic Press (1984) Gadamer, H. G. Philosophical Hermeneutics. University of California Press: Berkeley (1976) Bodner, G. M. Overcoming the Sports Mentality Approach to Program Evaluation: Action Research as a Metaphor for Curriculum Evaluation, A paper presented at the National Association for Research in Science Teaching Meeting, Atlanta, GA, April, 1993. TABLE 1, Survey Percentage Responses* Item Statem ent Agree Neutral Disagree AVG 1. I like using com puter sim ulations. 91 3 6 4.2 2. W hen using the com puter sim ulation, I worried that m y data would be lost, or the program would fail. 22 16 62 2.4 3. Tim e and budget constraints m ade the com puter experim ents m ore realistic. 86 3 11 4.2 4. The conventional lab experim ents worked better than the com puter experim ents. 5 11 84 1.9 -9- 5. The video tour of the plant added little to the value of the com puter experim ent. 30 30 40 3.0 6. It was easy to learn and operate the com puter sim ulation. 95 0 5 4.3 7. Com puter-sim ulation experim ents intim idate m e. 6 8 86 1.8 8. The speed of data acquisition in the com puter experim ents m akes m e uneasy. 6 5 89 1.7 9. Com puter experim ents allowed m e to focus on the principles to be learned, rather than the details of operating a particular piece of equipm ent. 82 10 8 4.2 10. Com puter experim ents are m ore interesting than conventional experim ents. 40 38 22 3.3 11. One disadvantage of the com puter experim ents is that I do not gain experience with the real plant equipm ent. 60 24 16 3.6 12. I would like to see m ore com puter-sim ulated experim ents in the chem ical engineering curriculum . 73 14 13 3.8 13. I would rather work on a com puter sim ulation because it is less hazardous than a conventional experim ent. 27 27 46 2.8 14. Conventional experim ents give m e a better sense of the kinds of problem s likely to be encountered in industry. 54 22 24 3.6 15. My group cooperated better during the conventional lab experim ents. 13 43 44 2.6 16. The design problem im posed by the com puter sim ulation is not as challenging as those encountered during the conventional experim ents. 30 22 49 2.8 17. A higher percentage of our tim e was spent planning the design of the com puter experim ents. 86 3 11 4.1 18. The com puter sim ulation allowed m e to study problem s that are m ore com plex and realistic than the conventional experim ents. 78 19 3 4.0 19. Com puter sim ulations allow m e to m ake m ore effective use of tim e by reducing the am ount of tim e needed to run experim ents. 95 5 0 4.4 20. The conventional lab experim ents were easier to learn and operate than the com puter experim ent. 6 8 86 1.9 21. Com puter sim ulations are a good way to learn new processes and concepts. 79 22 0 4.1 22. Com puter sim ulations work better than the conventional experim ents. 60 35 5 3.8 -10- 23. Com puter-sim ulations are m ore likely to “work” than conventional experim ents. 81 14 5 4.1 24. Overall, I think the present com bination of com puter sim ulations and conventional experim ents is appropriate. 57 14 29 3.4 *This table summarizes the results of two semesters. We have combined the Strongly Agree and Agree responses into one category: Agree. Strongly Disagree and Disagree have been combined into another category: Disagree. The Undecided responses are indicated as Neutral (N). -11-