Final Usability Test & Report

Deadline: Tuesday, May 13 by class time

Assignment Overview
This assignment should be read in conjunction with TA06 Final Interactive Prototype and TA08 Final Video. Together, TA06, this assignment (TA07) , and TA08 represent the culmination of work on your team project over the course of the semester. They also serve as a "final exam" of sorts as we will not be having a formal exam during our final exam slot (instead, we will be viewing your TA08 final videos).

While TA06 and TA08 focus on the specifications and expectations for your final interactive prototype and final video respectively, this assignment describes the usability test for your prototype and the required sections for your final report.

What To Do

  1. Begin by reading over TA06 Final Interactive Prototype and TA08 Final Video. Operationally, I suggest you assign "leaders" within your team to manage different components of TA06, TA07, and TA08 so that nothing falls through the cracks. While I expect that each individual member should contribute to all three assignments, I think having a specific leader will help organize the remaining tasks.

  2. Once you implement the three primary tasks for TA06, you must perform a usability test with three users. Ordinarily, this would be done with "target" users pulled from your expected user population; however, given the constraints of this class, I am loosening this restriction. Thus, you can use members of this class who are not on your team for usability tests (this is a rule change compared to the paper prototype test). It is an interesting experience being on the subject side of usability testing, and I'd like for class members to have a chance to have the experience. That said, I strongly encourage you to test with at least one non-CMSC434 student (and it's perfectly fine if none of your test subjects are classmates). Each user testing session must be done with at least two experimenters present (one to conduct the experiment and one to record observations/take notes). Before you conduct any test sessions, you must...

  3. Write a brief study plan. The study plan should include introductory content that you will read to each participant when they arrive at the test session, a description of your three primary tasks, a description of the type of data you plan to collect, and the interview/survey instruments you plan to use. For example, you should collect demographic data on your participants, which is often done via pre-study surveys. You should also give your participants brief post-study surveys inquiring about the effectiveness of your design and their overall user satisfaction. You will include this study plan in your report appendix. To help you with this, refer back to the Usability Testing reading assignment (RR9) and to the sample protocol I posted from my work at NCI. In addition, the user testing here will have a similar feel to TA05--only this time, your users won't be interacting with paper prototypes but a fully autonomous interactive prototype. :)

  4. After you write your study plan, test it out on a member of your team. This pilot test should follow the same protocol that you use for your three external users (i.e., it is a dress rehearsal for the real thing--try to make it as real as possible). You will include a section on how you conducted this pilot test, the results (if any), and how you changed your study protocol as a consequence (if you did). See "The Report" below.

  5. Now you're ready for the real usability tests. At a high level, each team should roughly follow this protocol:
    1. Recruit participants. You will have to describe your recruitment methodology in your report.

    2. As with TA05, download and modify this IRB "informed consent" template to fit your project [link]. Make sure you include a section about asking to "video record" the session. At the beginning of the user testing session, read the "Purpose of this Study" section of the consent form out loud to your participants. This should be done consistently for each participant. Then, give your participants a chance to read the entire consent form, ask questions, and, if they agree to participate, have them sign the form. If they do not agree to participate, simply wish them a nice day and recruit another participant (it can be slightly awkward but this happens!). If they do agree to participate, provide a copy of the form and take the signed copy for yourself.

    3. If the participant agrees to video recording the session, turn on the video camera. Do your best to protect the anonymity of your participant. Setup the video camera behind them (e.g., over their shoulder) so that you can capture their arms/hands interacting with the actual screens of your application. As with TA05, you can use anything to perform the video recording from a smartphone to a more sophisticated DSLR camera.

    4. After the informed consent process, you can begin testing your final interactive prototype. You might want to start with a brief "pre-study survey" asking basic demographic question (e.g., age, profession, computer proficiency). You will test each task in order. You can either read the task description aloud or have the participant read it him/herself. While executing the task, ask the participant to "think-aloud" (recall the think-aloud strategy we talked about earlier in the semester). One experimenter should record notes about how the participant is using the prototype, the problems/successes encountered, and any comments made during the testing session, as well as observations about non-verbal cues such as body language, facial expressions, and utterances. The experimenters should also record quantitative data such as how long each task took and the number of "problems" the user experienced. Include the qualitative and quantitative data notes for each participant in the appendix.

    5. After the three primary tasks have been tested, provide your participant with a short paper survey (called a "post-study survey") with a few questions inquiring about their overall impressions of the application and any specific comments for improvement. It's good to include at least one open-ended question where they can make any comment they want. Again, scan in and include these responses in the report appendix.

  1. After user testing, analyze your collected data (e.g., demographic survey data, observational notes, quantitative recordings, and post-study survey data). Again, see the RR9 reading (e.g., Section 10.4.8). Your report should include a participant table describing each participant (but not using their name--to preserve anonymity, you can refer to them by initials or first name and last initial, or as participant 1, etc.) as well as a task performance table (rows are participants, columns are task timings and "problems" experienced for each task). Focus your analysis on the best/worst performing design elements and think about how these findings can be used to improve future design revisions.

The Report

Submit a report of no more than 5 pages of text (approximately 2,500-3,500 words in a font similar to 11 pt. Times). Images, tables, figures, etc. are strongly encouraged, do not count against the page limit, and are thus effectively free.

Please construct your report with the following full sections and start each section with the header below (style Headings 1); it makes it easier for us to grade. So, for example, your first section should have the heading 1. Abstract and the next 2. Prototype Implementation.

Title Page

On your title page, please include a title (centered and bold), a catchphrase for your project that highlights its primary benefit, and your team name. Then make a table that lists each team member and describes their primary role/accomplishments for each of TA06, TA07, and TA08 (and TA09, the optional website, if you are doing that). If there's room, you may want to include some iconography or a screenshot of your app on the title page. Note: each team needs to assign an official proofreader that looks over the entire report, makes sure all of the requisite content is included, and eliminates typos/grammatical errors. Please include in the report who was assigned this role.

1. Abstract

The abstract should provide a 5-7 sentence (1/2 page) overview of your report including: (i) a description of your application, (ii) a description of your three primary tasks, (iii) a description of your evaluation method, (iv) a summary of your primary findings, and (v) design implications for future revisions.

2. Prototype Implementation

Describe your prototype implementation: what language did you use? Why? How did you build it? What libraries, if any, did you use? How were your design decisions informed? Did you rapidly iterate and test as you built the system? What interactions/features did you leave out but hoped to build? Justify your design decisions based on course content (e.g., lectures, readings, discussions).

3. Task Descriptions

Describe your three primary tasks (iterate from previous incarnations) This should be a four paragraph section. The intro paragraph summarizes the three tasks. Each subsequent paragraph should start with a style Heading 2 with the name of the task and a more lengthy description. The descriptions should include screenshots of your actual interactive prototype in the style of either: a sequential storyboard, a state transition diagram, or a branching storyboard (see this Greenberg reading).

If the tasks changed from the previous assignment, please describe these changes and why they were made (e.g., was it based on your own design reflections, was it based on the experiences of TA05, etc.). You can also reference your TA08 Final Video here (this is called a video figure rather than a pictorial figure and is becoming increasingly common). So, if you have a skit in your video that goes over each of the tasks, you can refer to the video timeline in your prose (e.g., See 01:20-01:44 for a video describing Task #2). This should not replace your prose but should augment it.

4. Usability Tests

This section should have four sub-sections focused on the following. Each sub-section should be a Heading 2. See the RR9 "Usability Testing" reading.
  • Describe how you pilot tested the interactive prototypes amongst the group (or friends/family) and the resulting changes that were made either to the study protocol or to the final interactive prototypes themselves.

  • Describe how you recruited participants and their demographics (relevant to the project).

  • Describe your study method, which includes how you performed your study including the study protocol, the location, the length of the study, and a description of the data collection instruments (e.g., the video recording setup, the post-study survey).

  • Describe your analytical method, which describes how you analyzed the data you collected.

5. Results from User Testing

This section should detail the primary results from user testing and implications for design.

6. If We Had To Do It All Over Again...

I want you to reflect on whether you think the iterative design process we used this semester actually improved your final interactive prototype. Why or why not? What would you do differently if you could? Do you think it would be better to try and code up simple designs early (say, starting from week 1) and focus on improving these?

7. Video Making Process

Include approximately 2-3 paragraphs on your video making process for TA08. How did you divide up the work? What editing software did you use? How did you decide on your particular approach? How did you record the actual screen interactions? Note: Because this text is part of TA08, it does not count toward the 5-page limit.

The appendix should include the following. Each of the following bullets should start on their own page and should be titled Appendix A: <title>, Appendix B: <title>, etc. in Heading 1 style.
  • Your study plan
  • Raw notes from the three user testing sessions (notes should be clearly marked with a timestamp and session number)
  • A scan of the post-study paper survey responses (these should also have session numbers)