Usability Testing a Chemical Reaction Database Website

Usability Testing a Chemical Reaction Database Website A thesis submitted in partial fulfillment of the requirements for graduation with Honors in Computer Science. 2022 Table of Contents Abstract ............................................................................................................................... 1 Introduction......................................................................................................................... 1 Background ......................................................................................................................... 2 Participants and Procedure.................................................................................................. 5 Participants...................................................................................................................... 5 Procedure and data collection ......................................................................................... 5 Interpretation................................................................................................................... 5 Results................................................................................................................................. 6 Missing References ......................................................................................................... 6 Unclear Formatting Requirements .................................................................................. 6 Confusing Advanced Search Conventions...................................................................... 7 Previous Searches Lost ................................................................................................... 7 Reliance on the Basic Ligand Search.............................................................................. 8 User Suggestions............................................................................................................. 8 Other Suggestions ........................................................................................................... 9 Discussion ........................................................................................................................... 9 Recommendations............................................................................................................... 9 Conclusion ........................................................................................................................ 10 Appendix A – Test Script ................................................................................................. 11 iii Abstract This report shares the results of user testing a new website to host the data from the NIST-46 database of chemical constants. This database contains critically selected stability constants of metal complexes and is mainly used by professional chemists in academics and research. I discuss the test background, procedure, and findings, and assess the issues and suggestions that were encountered as well as possible solutions to address these concerns. This report should provide some guidance for the next team to work on this project. Introduction The data contained in the NIST-46 chemistry database has been incredibly tedious to access and navigate for chemists and is no longer maintained. The goal of my team’s senior capstone project was to develop a website that would allow chemists to quickly and easily access the chemical reaction constants they are looking for. To design the website, we consulted our client, Associate Professor of Chemistry Nate Boland. My team and I built the website, and I conducted user testing to generate feedback and identify areas of concern. This was a good time to conduct user testing as the website was believed to be fully functional. It is beneficial to receive user feedback before much more work on the frontend would be done, as it would set priorities and inform design decisions. My work with test participants provided insight into the usability of the website. This type of study is known as a usability study or user study and falls into the broad area of user experience or human factors evaluation.1 Because of its formative role, this study is exploratory rather than hypothesis-driven and qualitative rather than quantitative. This paper is divided into six sections. The background section details how the website currently functions, providing the necessary context for the reader. Next, I explain how the participants were collected and testing conducted. Then I outline my findings from the user testing, detailing concerns and possible solutions. I discuss trends and common themes from these findings. In the recommendations sections, I rank my suggested improvements to the website. I conclude with a summary of my report. 1 Moran, K., 2019. Usability Testing 101. Nielsen Norman Group. Retrieved from https://www.nngroup.com/articles/usability-testing-101. Background The source code for the website can be found at two GitHub repositories. The frontend source code can be found at https://github.com/WhitmanCSCapstone/nist-46 and the backend API source code can be found at https://github.com/leilapetersonwhitman/apace-api. As stated previously, the new website is connected to a database that contains the data from the NIST-46 database. The website in its current state allows users to search for critical values of metal complexes. The chemistry-specific terms are not important for the reader to understand; they can be understood as different possible search parameters or parts of result data. The site provides the user two options for searching: ligand search and advanced search. The home page defaults to ligand search, which can only take in a name (full or partial) of a ligand and displays results as unique ligand-metal pairs, see Figures 1 and 2. If a user clicks on an entry, they are taken to the data page for the pair, which contains the constants users look for (Figure 3). Figure 1: Ligand search Figure 2: Ligand search results 2 Figure 3: Constant data page The advanced search takes in any number and combination of parameters out of the five options (category, ligand, metal, metal charge, and ligand charge) and uses those to refine the search, shown in Figure 4. Figure 5 shows the results of an example advanced search query. Like the ligand search, if a user selects an entry, they are taken to the appropriate data page (Figure 3). Figure 4: Advanced search 3 Figure 5: Advanced search results At the end of my capstone group’s time working on the project, we discovered that the data in our database did not match the previous database and was not accurate. While this is a hindrance to the website being published in its current state, I decided to continue with this user testing project as it is relatively unaffected by this data issue. 4 Participants and Procedure Participants I recruited a total of four participants for the study. This number of participants is sufficient because testing with around five participants lets you find almost as many usability problems as you would find using a much larger number of participants.2 The website also has a relatively small number of overall target users, so a very large sample size was not necessary. The participants were all professional chemists as they are the ones who will be using the website once it is finished. The participants included three professors and one graduate student, who had all previously used the old NIST-46 database. Procedure and data collection Before testing was conducted, I received approval from the Whitman College Institutional Review Board (IRB). Each session lasted at most 30 minutes and involved one researcher (me). Before the participant’s testing slot, they read and signed an informed consent form. The website was tested by each participant separately. All sessions were conducted over Zoom. I interacted with the participant, following a study script (Appendix A). The participant was sent a link to the website, which they navigated to and then shared their screen with the researcher. Before asking participants to complete the sample tasks, I asked some questions about the participants’ experiences with the existing NIST-46 database, to help put the participants’ responses in context. Then I asked the participant, one task at a time, to complete a total of three sample tasks using the website. In this case, all tasks involved finding data for an entry that fit a specific set of parameters. The participant was directed to think aloud while interacting with the website on their own computer. After all the tasks were completed, I asked some questions to further assess the usability and desirability of the application. I took notes on the participant’s words and actions. All the sessions were recorded; the recordings capture audio and the participant’s screen. This allowed me to go back after the sessions and take more detailed notes and observations. No revisions to the website were made between participants. Interpretation I informally reviewed my notes and recordings to identify usability problems and opportunities for improvement. I used a spreadsheet to help organize and group observed issues. I did not collect any quantitative data or do any formal statistical analysis. 2 Nielsen, J., 2012. How Many Test Users in a Usability Study?. Nielsen Norman Group. Retrieved from https://www.nngroup.com/articles/how-many-test-users. 5 Results In this section, I will discuss my findings from the user tests. Each subsection will outline an issue that was encountered by the participants, the severity level of that issue, and my recommendation for solving that problem. The subsections are arranged in order by severity of the issue, from most to least severe. The issues are missing references, unclear formatting requirements, confusing advances search conventions, previous searches lost, and reliance on basic ligand search. Suggestions are also included at the end of this section. Missing References A non-procedural issue that most participants commented on was the lack of references for the data. My capstone team has been aware from the beginning that this was important data that we were lacking, but the issue is important to include in this report. When discussing the missing references in the post-task questions, one participant said that since the database has not been updated since 2004, they would use the references in the old database to find newer data. They did this by finding what they were looking for in the database, finding the reference to the paper it was originally published in, and then using a scholarly website to find other more recent papers that cited the original paper to find more up-to-date data. This response also supports the need for the ability to submit new data to be included in the database, which would be a new feature to add. Another participant stated that different references can give different values for the same reaction, and they would want to be able to use their own judgement between the sources. The clear solution to this problem is to parse the references from the extracted data files and include it in the new database, and then display this information on the website. This is a high severity and high priority issue. Unclear Formatting Requirements A prominent issue was participants inputting values in search fields in formats different than the one(s) required, and thus the search not working. There were seven instances of this, among three out of four participants. All three of these participants experienced confusion over the ligand name search, thinking that they had to find the value for the ligand with the full name of “alanine” or thinking they had to come up with a ligand to search for that contained “alanine” in the name, such as “glycylalanine,” instead of using it as a partial name search. This could be due in part to the phrasing of the task— “Please find the value of a constant for a ligandmetal pairing with a ligand name that includes “alanine””—which could be interpreted in multiple ways. Regardless of the test procedure, I believe it would be beneficial to include text somewhere among the ligand search that states that it can search for partial names. More specific formatting errors came up with searches involving different parameters. As shown in Figure 4, the advanced search option lets a user search by any combination of category, ligand, metal, metal charge, and ligand charge. Users were tasked with searches involving metal and metal charge. For the metal field, which takes 6 the element symbol with the first letter capitalized, participants searched for the symbol uncapitalized, the element’s full name, or for both the element symbol and charge in the same search field. Similarly, for the metal charge, a participant tried to search for “3+” when the search query only takes +number, –number, or the number alone. In some cases, the results panel told the user that there were not results for that query, but for other inputs the results panel remained completely empty, not providing any user feedback. It would be helpful to add feedback to the user telling them if they might have input invalid search terms and/or if their search failed to execute because of invalid search terms. In both, participants attempted to fill in the search fields with values the query engine would not accept. I believe this issue can be resolved by including more instructions and formatting guides with the search fields. These instructions could include placeholder values that show the correct format and/or guides on the formatting requirements of the search term located above or below each unique search input field, or as hover text. An additional solution would be for the search query to accept a wider range of inputs, such as all lower-case metal symbols or a different order of +/- and number for charges. I would classify this as a high severity, high priority issue. Confusing Advanced Search Conventions Another issue I observed with three out of four participants was confusion over how advanced search worked. Two participants did not seem to understand that they could search by more than one parameter using the advanced search tool, and one thought that they had to fill in every category of search parameter. The tool actually works with any number and combination of search parameters offered. The solution to this confusion could be including a line of text somewhere above or below the list of possible parameters stating these use conventions. I would classify this as a mid-high severity issue. Previous Searches Lost A less obvious but still important issue is that the website currently does not remember any previous searches. All the participants experienced this and expressed surprise, but only two actually commented on this issue. When a user clicks on a data entry that takes them to the page for that ligand-metal pair, they can’t easily go back to check out another pairing with the same search parameters. Clicking the back button takes the user back to the home page and empty ligand search, even if they were previously using the advanced search. This would become frustrating if someone was trying to look through multiple entries with the same parameters. My suggested remedy would be to implement some solution that keeps track of the page navigation and would allow users to go back to the page they were just on. An appropriate remedy requires further research, as there are multiple ways to do this. I would classify this as a mid-severity problem. It does not inhibit use of the website but would quickly become frustrating for users. 7 Reliance on the Basic Ligand Search Three out of four test participants, when asked to complete a search with parameters beyond a ligand name, chose to conduct their search using the basic ligand search option that has the limited capability of only refining by ligand name. My assumption was that for these more complex searches, the users would explore the advanced search tab option and use it to refine by additional parameters such as metal and metal charge. When looking at why most participants did not choose that route, I looked at both the visibility of advanced search and the construction of the user testing procedure itself. As seen in Figure 1, there are two search options presented to the user, Ligand Search and Advanced Search. The home page of the website defaults to Ligand Search, and the current search mode is made bold. More specific user testing could be conducted to determine if advanced search is a clear option to the user. This phenomenon could also be caused by the testing procedure. As seen in Appendix A – Test Script, the first assigned task is to find a value for a pair in which the only parameter is a partial ligand name, which would bring users to the ligand results shown in Figure 2. Here users can see the unique metal options included in these results. Thus, when asked to complete the second task of finding a value for a pair with both ligand and metal parameters, they would know that they could find the metal in the results just by searching for the ligand name. The knowledge of this valid way of searching could discourage them from checking for an advanced search option. More testing could be done in which the order of assigned tasks is changed to determine if advanced search is a clear and intuitive option for the user. Along with this testing, questions could be asked regarding how users think increased familiarity and frequency of use would affect their use habits. Depending on the underlying cause of this issue, I would consider this a low to mid severity problem. User Suggestions One of the questions in the post-test interview was if the participant had any suggestions for further improvements. In this section I consolidate their suggestions. Multiple participants suggested additional search parameters. Two participants expressed interest in searching by chemical formula, with one explaining that sometimes naming conventions are unpredictable whereas formulas are not. If a user did not know the exact name of the ligand they were searching for, they could search by the formula to find the name. Also suggested were searching by reference author and searching by chemical structure drawing. The participant who suggested structure search acknowledged that it would be a huge project, but it is what many chemists are most familiar with and recognize. I would prioritize adding searching by formula as it likely would not be too difficult and seems to be a popular request. Another suggestion was to include drawings/diagrams of the ligand structures. This is a feature that the old database had but is not yet included on our new website. The drawings are important for chemists to recognize what compound they are looking at, as the name might vary, and so should be included. There was a request for clearer conventions to help people who are not as familiar use the website, which is expanded on more in the above section. With this, one 8 participant proposed having a master list of categories somewhere on the website, as the inclusion of ligand categories and the ability to search by them is a new feature not present in the old database. This would help users know what they could search for and use the search tool more effectively. It would also address a consideration that came up with a different participant, who mentioned that in the old database, ligands were numbered sequentially in way that resulted in ligands with similar structures next to each other numerically. It was useful to be able to easily find structurally similar ligands, which is hopefully also what the category search could accomplish. There was also feedback that the search by ligand charge was a little confusing, as it would be more appropriate to call it something else, possibly related to protonation or the ligand’s neutral form. I would suggest consulting with the client or gathering more user feedback on this because of the domain-specific knowledge it requires. Other Suggestions There are some ideas for improvement that I thought of during the user testing but that none of the participants explicitly commented on. One is the ability to sort the results by the different categories/parameters, as it would help users find what they were looking for faster. The other thing that could be improved is the formatting of the notes/footnotes. Currently they are contained within curly braces and quotations which do not need to be there. It would improve user readability to have them in a better format. Discussion There are two major trends to be noted in this investigation. The first is that a large number of the issues encountered come down to a lack of clear instructions. The goal of my group’s capstone project was first and foremost to get a working website that could perform queries and serve data running. Including thorough instructions was a lesser priority that we did not get to in the time we had. But now that the website is mostly complete, adding instructions and guides would be a fairly easy task to accomplish with a large payoff for the users. Another recurring consideration is the uncertainty about whether something is fundamentally a problem or the users behaved the way they did because of the way testing was conducted. More testing, with possibly different groups doing slightly different procedures, or more thorough testing could help clarify this confusion. I would recommend waiting until some issues are addressed before conducting more user testing, as the most pressing problems are clear from this first usability test. Recommendations Here I rank the list of possible changes or issues to address first in order of the severity of the problem, what is most important to the usability of the website, and second in order of presumed ease with which the change can be made or problem addressed. These are my recommendations to the next group that works on this website. In order of severity: 9 1. Parse and add references 2. Improve user instructions and help a. Include text instructions, help buttons, hover text b. Add placeholder text in search fields of acceptable example inputs 3. Implement a solution to remember previous searches and pages 4. Change frontend and/or API to accept wider range of input a. Capitalization agnostic b. Metal charge symbol placement 5. Provide either a dropdown menu or master list of categories 6. Implement formula search 7. Include chemical structure diagrams 8. Consult client and potentially rename ligand charge search 9. Implement result sorting capability 10. Improve formatting of notes In order of estimated difficulty (low to high): 1. Improve user instructions and help 2. Consult client and potentially rename ligand charge search 3. Provide either a dropdown menu or master list of categories 4. Improve formatting of notes 5. Implement formula search 6. Implement a solution to remember previous searches and pages 7. Implement result sorting capability 8. Change frontend and/or API to accept wider range of input 9. Include chemical structure diagrams 10. Parse and add references There are a few interesting observations to be made about these lists. One is that the first item on the severity list is the last on the estimated difficulty list. Conversely, the second item on the severity list is presumably the easiest issue to solve. Most of the remaining tasks are not so polarized or parallel. It will be up to the next group to decide what to prioritize based on this knowledge. Conclusion The user testing that I conducted illuminated a number of issues with the website as well as ideas for improvement. Some of these issues are quite severe and should be addressed as soon as possible, such as the lack of references and confusion over how to use certain search modes or input. Others are not as critical but would be useful to the user or improve their experience on the site, like adding more search parameters or including structure diagrams. I hope my findings will be of use for the next group that continues this project. 10 Appendix A – Test Script Pre-test disclaimer: Just so you know, it has come to our attention that there were errors in the data we received from our student colleagues to populate this website is not all correct. Please disregard the data itself, we are testing the design of the website only. To that end, we are testing the website, not you, there are no wrong answers. Pre-task questionnaire 1. Have you used the NIST-46 database before? 2. If so, how often do you use it? I will now send you the link to the website, please open it in a browser and then share your screen. I will now ask you to complete some tasks using the website. Please think aloud while interacting with the website. Your tasks 1. Please find the value of a constant for a ligand-metal pairing with a ligand name that includes “alanine”. 2. Please find the value of a constant for a ligand-metal pairing with “hydroxylamine” as the ligand and H as the metal. 3. Please find the value of a constant where the ligand name contains “ether” and the metal is Fe with a charge of 3. Post-task questionnaire 1. Were the tasks representative of how you use NIST-46? If not, can you suggest a more typical task? 2. Would you rather use this than the old NIST-46 database? Why or why not? 3. Do you have any suggestions for further improvements? 11