Focus Testing Basics

CMS.611J/6.073 Fall 2014 Focus Testing Basics Key Points: Choosing focus testers; what to know before you start testing; advertising; what do your testers need to know before they start playing; observing testers and gathering data; treating your testers with respect; What do you do with the data; Quick Checklist of Things to Do and Have. Choosing Focus Testers:  Know your audience.  Know what you want to test. This is most relevant for out of class testing, as where you advertise and who you contact affects what kind of person attends. For in-class testing, you're going to get a fairly homogenous group of players: college students with a strong interest in video games. Most of the people in the world aren't like that, so you want to keep that in mind for your results. What kind of people you want as your focus testers depends on what your goal is. If you're trying to create a game that you think only hardcore, serious video game players will play, you're going to want to get people who spend thirty hours a week playing videogames, and the other ninety hours a week thinking about them. If you're making a game targeted at ten year olds, you'd better get some ten year olds in to test it, because it doesn't matter how much college students like it; that's not your target audience. On the other hand, if you're trying to create a game that appeals to a broader section of people, you want a broader selection - folks who play games a lot, occasional players, people who usually sit and watch someone else play, but can be tempted into playing 'just one game.' They are harder to find, but that makes their feedback more valuable - because they are less informed on how to play video games, and less skilled, they represent a ' casual player' better - because they ARE. They often have more problems with the UI, or understanding instructions, or slower reactions so the games are just too hard for them - and for a potential 'casual' game, that information is really important. Advertising for Focus Testers:  Advertise for your audience in advance.  Choose the appropriate channel.  Give them a reason to come.  Your target tester isn't going to walk in the door just because you need them. So, think about where they are likely to hang out, and what communication method will get their attention. You also need to give them enough time to make it to your testing site, but don’t advertise so far in advance that it gets forgotten by the time the actual event rolls around. Do you want casual gamers? How about postering at an Internet café, or a coffee house that offers wireless? Do you want college students? Poster on campus, or in dorms. 1 CMS.611J/6.073 Fall 2014 People have inertia; by default, they aren't going to come. If you can offer them something -even something as simple as pizza and soda, or cookies - it may increase the likelihood of them appearing. Giving people an email address to RSVP to may also increase response; after all, people who have responded feel a bit of commitment. What do you – and your testers – need to know before you start testing?  Know your focus test goal.  Be prepared with questions or surveys for your players.  Be prepared with scripts for the team running the test.  Baseline questions. First, and most important: why are you focus testing? To determine your game’s overall playability? Level of difficulty for a new player? A specific mechanic implementation? Ease of use with your current user interface? Have a goal – a key goal – expressed as a question, preferably in simple English. “We want to know if a new player unfamiliar with rhythm games can understand and start playing our game quickly.” Secondly, you need to know what information to share with testers before start playing. In a perfect universe, you could set them down in front of the game and start them playing. However, frequently user tutorials, clear UI, and helpful hints are among the last part of a game to get in. So how do you explain the game to your player, giving them the same amount of information they would get from the game ‘in the real world’? By writing and using observer scripts for the people running the prototype, and preparing player handouts for your testers. Player handouts are a single sheet of paper, with all the relevant keys, instructions, and ‘how-to’ that you expect to give your players in your final game. Think of it as a reference card – if a player wants to know how to ‘jump’, they should be able to check their handout, and not need to ask a hovering observer. Observer scripts are written for the people running the focus test. They list what information players should be give before starting the game, what information players should receive after playing, and what questions the observers should ask (and when to ask them.) By standardizing the information you give your players, you have a better chance of getting usable data from them. Scripts also control for the natural tendency developers have to ‘show-off’ their game. If you leap in and ‘show’ your testers what they ‘should be doing’, you will never learn what your players really will do, without a helper to point things out. Unless you intend to ship a team member with each game, that’s not going to be helpful. And, finally, you need to know what you want to know from each player. Is observing them and taking notes enough? Do you have a list of specific questions you want answered? How are you going to gather – and record – the data from the focus test? Think about whether you want a interview with the observer and set of questions to ask; a one page survey, or just notes from the observer watching the game being played. One more thing: baseline questions. Sometimes, you want to know how experienced someone is, so you can take their results with a grain of salt – if you are testing a physics learning game, you are going to get a very different understanding of the game from a fifth grader vs physics teacher. So you may 2 CMS.611J/6.073 3 Fall 2014 want to ask some basic baseline questions about your testers. Try to *only* ask these questions if you actually need the data – every question you ask about the player is one less question (generally) you can ask about the game, since each player has only so much patience for answering questions! Sample baseline questions include:  How many hours a week do you play games?  Do you consider yourself a hardcore gamer?  What is your favorite kind of game?  What age group are you (15 and under) (15 – 25) (25 – 35) (35 and up)  Do you like ____________ type of games? Observing Testers and Gathering Data  Have instructions for the game ready  Observe without interfering or leading your players.  Keep surveys short – one page max! You should try to keep your focus testing as scientific as possible: while your test is running, you don't want to contaminate the data. In this case, that means avoiding influencing your testers, or ruining your results by giving them outside information. In practice, this means you need to know exactly what information it is reasonable for an 'average player' to have, and in what format, and to give that, and only that, to your tester. So, if your game is supposed to be downloaded from the Internet, and have all of its instructions in its menu system, you want your testers to receive all of their information from the game - not their observers. Ideally, you should sit back quietly, watch them play, and take notes; if they ask for more information or 'hints', you should shake your head, and ask them - for the sake of the testing - to make do with the information available to them. If there is an unexpected problem with the game, or testing, then you should intervene. The game crashes, a new mechanic isn't working properly, etc. After the testing is done, you will probably have a few more questions to ask them; those can either be asked verbally, and the notes written down (if so, have just a few questions); or, you can hand them a questionnaire to fill out. Keep "time spent filling out paperwork" less then "time spent playing the game"; for short games, more than one page of questions is too much. Treating Testers With Respect:  Don’t tell them what to do; ask them what is wrong.  If you notice a player is frustrated, ask them why, and ask them if you can help, or if they’d like to quit.  Let players fill out their surveys in privacy.  Don’t ask personal questions. Your players are helping you make your game better, so treat them well. Be courteous, and make sure that they feel comfortable. They've decided to use some of their time to help you with your work-inprogress, and it is their choice to be here at all times. Remember that your testers may quit testing at any time, and that answering questions is voluntary - they may choose to answer some, all or none of the questions asked of them. CMS.611J/6.073 4 Fall 2014 Also, while you should be as much of a 'silent observer' as possible, remember that you are observing a person. If the tester is clearly getting too frustrated, too annoyed, upset or angry, ask them about it. Ask them what is so frustrating, and ask them if they would like assistance. Note that your help was necessary - because no game should make someone that frustrated - but give them a hand when needed. After Testing: Now What?  Review your observations and your surveys.  Decide how to use the results.  Review your survey and focus test goals, so you can improve the process next time.  Remember your testers are TESTERS, not game designers. Once you've finished the focus test, gather up your notes, and the team, and review the information. Once again, it really helps to have a couple of key questions you are trying to get answered "Is the game too hard?" "Is the UI easy to use?" "Is the game fun?" Look at the data through those lenses, and see what you are doing right – and what you are doing wrong. You will often learn things about your game you didn’t expect to, things that you didn’t even realize were a problem, or things that the players enjoyed you hadn’t expected them to. Take advantage of that! On the flip side… sometimes the answers are all over the place, and you can’t quite figure out what to do with it. Review your surveys, think about how you could focus your questions a bit better. And, finally – players will always be happy to tell you ‘what to do’. They will have suggestions, ideas, complaints, you name it. Accept those suggestions and complaints politely and gracefully – and then, think seriously about throwing them all out. Just because one player – or ten - wants you to add guns to your game doesn’t mean your game needs guns; it may mean that those players are ones who couldn’t imagine playing a game without guns. Possibly you need a group of less aggressive focus testers. In the end, you are making the game, and the final design decisions must be guided by you, your team, and whatever the overall goal of your game is. Your focus testing data is just that – data. You need to decide how to interpret it, and how to use it. Quick Focus Testing Checklist. (Not intended to cover all situations!) Pre-Focus Test 1. Decide on the focus test's goal - the best goals are formed as a question that your observation of testers will answer. "Do people enjoy the game?" "Is our UI easy to use?" "Do people understand our life mechanic?" 2. Choose your audience. 3. Choose a place and a time (should be convenient for your target audience!) 4. Write up a set of questions (if you want a filled out survey.) 5. Will the testers need any additional information to play your game that isn't included in the game? You may want to write up a simple How To Play guide, if your game isn't advanced enough yet to have necessary instructions embedded in the shell and UI. CMS.611J/6.073 5 Fall 2014 6. Make sure you have a stable working build, installed and tested on each of the testing machines, that will allow your testers enough functionality to answer your questions. During the Focus Test 7. Have at least one observer for each station where the game is being played. Observers should take notes, assist with starting and ending the game, provide surveys for filling out afterwards, and answer any questions. 8. Ideal observers? Everyone on the team! Designers, programmers and artists all see the game a bit differently, and likewise often focus on different aspects of the game. 9. It's often helpful to have one central person collecting finished surveys, and any notes from the observers that go with the surveys; that way things don't get lost. 10. Don't ignore observations just because they don't match your Questions. Testers will often turn up problems or make observations that never occurred to the team to think about. After the Focus Test 11. You need to have a meeting to discuss the results, and decide what if anything the team is going to do about it. You can either go over the raw data (read the surveys out loud together!) at the meeting, or have someone compile the survey and observational data into a reasonable synopsis. (Having someone compile the data into a reasonable synopsis is a good time saver, but there is also value in everyone seeing the raw data, thus avoiding single viewpoint bias.)