Image

Digital prototypes: evaluation results

In the past week I performed the evaluation of the digital prototypes with 5 users. The goal of these evaluations was to determine which prototype was preferred by users, to see if there were any major usability problems left and to get an idea of the perceived usefulness of the app.

The results are very conclusive, 5/5 users found the 3rd option of the first new method the best one. This means we will be using this to develop a fully functioning app which enhances your email/SMS/Whatsapp/ Slack/Whatever’s-possible notifications with our app!

The entire report can be found here, along with the Excel sheet used for the scenario: evaluatie3Scenario.

Final AWARE results

After receiving the results of the last people who had installed the AWARE framework, I added them to the data collection to take another look at the resulting graphs. Unfortunately the AWARE database file I got from some people wasn’t working, it was corrupt. I googled the problem myself first and couldn’t find any solution, I tried opening the files with several database managers but (obviously) none were able to open the file. So while studying and taking yet another break, I decided to send an e-mail towards the developers of the AWARE framework asking if they were aware of this problem and if they maybe had a fix. Turns out they did know and forwarded me this link: http://random.kakaopor.hu/how-to-repair-an-sqlite-database, which was quite an easy fix but somehow this never showed up in my search results. Guess after 5 years of Googling everything, I’m still bad at it.

So having fixed all the database files, I now have data of 14 students. These consist of 13 males and 1 female student. 9 are computer science students, of which 1 Phd student. As we can see in the graphs below, the most used apps are still very similar to our intermediate results. The chat app of Facebook by far generates the most notifications, which results in a lot of micro-usage sessions and a lot of longer sessions as well.

Final AWARE results - graph 1

We can clearly see most of the apps are still communication apps, and a lot of micro-sessions are occurring. So I think we are still justified in our new direction of enhancing notifications to allow for handling them quicker, which hopefully could reduce the amount of micro-sessions.

The graph of session duration still looks more or less the same as the one from the intermediate results. It is obvious most user sessions are still 0 – 20 seconds.

Final AWARE results - graph 2

Evaluation digital prototypes

This week I did an evaluation of the digital prototypes with 5 users. The full report can be found here: evaluatie 2.

From these evaluations I can conclude that there are few usability problems with the applications. The only real problem remaining is the mute button of the class assistant app which should be a toggle button instead of an action button.

Although most people perform micro-usage very often, this is mostly when checking their calendar for appointments, viewing notifications or checking social media. Very few people use their smartphone to help them study or create shopping lists. Therefore the users indicated they would only use the class assistant app frequently.

Paper prototype evaluations

After my weekly appointment with my supervisor on Monday, my task was was to create a paper prototype for 2 other applications and evaluate them with 5 students. I chose to create a paper version of the cooking assistant app and the Pomodoro clock. I also slightly changed my class assistant prototype since I couldn’t use the ‘sliding window’ prototype as explained in my previous blog.

TemplateIn case any reader wants to create a smartwatch paper prototype, here is the template I created, containing an LG G watch in true dimensions and lots of squares to draw in. (Source watch image)

I’ll start off with an explanation of the paper prototypes. Next I’ll show and discuss the results of the evaluations.

2014-10-26 10.53.03 copy

The first one is the Pomodoro clock or the study assistant. For those who don’t remember, the Pomodoro technique consists of concentrated studying for e.g. 25 min, followed by a 5 min break. The purpose of this app is to quickly allow you to start up the timer and provide a watch face where a quick glance is enough to see how much longer you have to study. I numbered the screens in the image above to easily reference them.

It’s 9 o’clock and you want to start studying (0) so you start up the Pomodoro app on your watch (1). I left out the details on how to start this app since there are multiple ways to do this and it would just make it confusing for the people evaluating who are not familiar with the Android Wear platform. When pressing the “start working” button, the screen switches to the clock view (6) and the minutes start counting (7) and a red/pink line traces the passed minutes. The end of your work session is indicated by the green marker. Alternatively, the user is presented with his regular clock and a notification card (9) which indicates the time he has been studying.

When swiping either the clock or the notification card to the left, the user sees the “stop” button (2). Another left swipe shows the Duration setting screen (3), where the user can change the duration of his work and break sessions. Another left swipe presents the user with the option of displaying either the elapsed time or the time left of his session on the clock. (8) shows what the analog clock would look like when the time left option is chosen.

2014-10-26 10.55.27 copy

The second app is the class assistant app. The purpose of this app is to remind you just in time when you’ve got a class and quickly present you its location. When the lecture starts it provides you with quickly accessible actions.

Say you’ve got a lecture of the statistics course at 14h. 20 minutes beforehand you get a notification (1). Swiping this notification up presents the full ‘notification card’ and reveals more information (2) such as the starting time and location/classroom. Alternatively, the ‘peek notification’ already shows you all of the information (3) and swiping up only enlarges this information and also shows screen (2).

When the lecture starts, you receive another notification (5). Swiping up reveals the entire card again (6). Swiping the card to the left (works both in 5 and 6) you are shown the ‘Mute phone’ button (7). Pressing this button mutes your phone and shows (9). Another left swipe shows the ‘Record audio’ button, which starts recording audio when pressed and shows screen (10). When pressing the ‘Stop recording’ button, a confirmation is shown for a second or two (11) and the screen returns to (8).

2014-10-26 10.58.04 copy

The third and last app is a shopping and cooking assistant. The purpose of this app is to help you during shopping and cooking (surprise surprise). We imagine a smartphone app which contains a lot of recipes and has a button ‘view on smart watch’ or something similar, which results in a notification card showing on the smart watch (0).

Swiping this notification card to the left shows the “Shopping list” button (1). Pushing this shows all the needed ingredients with check boxes (3). Optionally, when ticking off something, it moves to the bottom of the list (5). Another possibility would be to show a ‘peek card’ (6, 7) with the possibility of immediately checking off the first item on the list by pressing the blue check mark. After ticking off the last item on the list, the user returns to the clock with the notification still active (0).

When returning home two swipes to the left will result in the “Recipe” button (2). Pressing it shows you the first step of the recipe (8) and swiping it to the left shows the next ones (9, 10). Swiping the last instruction to the left shows the “Klaar” (should have been “Finished”) button (11) and pressing it closes the app and returns you to your regular clock (12).


Evaluations

So these are the applications I introduced to my evaluating users. The ‘think aloud protocol’ was explained and used and every session was recorded. I will present and discuss the results of the evaluations.

Evaluation 1 – Computer Science student, 23 years old, male

Application 1 – Pomodoro

  • Analog clock is clear
  • To stop working, thought he had to push/tap the clock. Next he tried swiping it, which was correct
  • Digital clock/notification card is a bit less clear
  • Likes the available settings
  • Proposes to add a setting where the user can choose between the digital and analog clock

Application 2 – Class assistant

  • Likes the alternative notification (3), if the presented information can be kept concise and readable
  • Confused the “Mute Phone” button for a toggle button. I.e. he thought his phone was muted when you can see the “Mute phone” screen (7), and unmuted when you can see the “Unmute phone” screen (9)

Application 3 – Shopping/cooking assistant

  • Likes the alternative where checked off items move to the bottom of the list (5)
  • Thinks it is good to return to the clock after ticking off the last item of the shopping list
  • On the recipe steps where you have to time something (e.g. cooking pasta (8)), he would like a small button to quickly start up a timer
  • The ‘peek card’ shopping list is not good since you usually don’t check off a list from top to bottom in order while shopping

Evaluation 2 – Medical student, 22 years old, male

Application 1 – Pomodoro

  • To stop working, thought he had to push/tap the clock. Next he tried swiping it, which was correct
  • Prefers the analog clock
  • Proposes a different UI instead of the clock: a circle which represents the full time you have to work and a pointer which instead of tracing the passed minutes, indicates the percentage of time completed of the work session.

Application 2 – Class assistant

  • Prefers the simple notification (1), since after a couple of weeks it’s likely you know the location by hearth
  • Likes the class actions (mute and record) (7, 8)

Application 3 – Shopping/cooking assistant

  • The ‘peek card’ shopping list is not good since you usually don’t check off a list from top to bottom in order while shopping
  • Proposes to add a button on a screen next to “Recipe” (2) with a timer button

Evaluation 3 – Computer science student, 22 years old, male

Application 1 – Pomodoro

  • The function of the green indicator wasn’t immediately clear (7)
  • To stop working, thought he had to push/tap the clock. Next he tried swiping it, which was correct
  • Prefers analog clock
  • Likes the proposed possibility of also being able to choose between the analog and digital clock in an extra settings screen

Application 2 – Class assistant

  • Prefers the simple ‘peek notification’ (1), but would like to be able to choose which information can be shown
  • Suggests adding the ending time of the class on the card (2) and extending the location information with the full address of the building
  • When recording (10), would like an indicator somewhere to show that you are still recording, or a timer indicating the length of the recording so far

Application 3 – Shopping/cooking assistant

  • The ‘peek card’ shopping list could be useful, if the list is ordered logically according to the grouping of the store
  • Does not like the alternative where ticked off items move to the bottom of the list
  • Proposes an extension of the shopping list, which could be used when buying books for classes

Evaluation 4 – Pedagogy student, 22 years old, female

Application 1 – Pomodoro

  • To stop working, thought she had to push/tap the clock. Next she tried swiping it, which was correct
  • Prefers analog clock but digital version might be useful as well
  • Likes the proposed possibility of also being able to choose between the analog and digital clock in an extra settings screen
  • Not immediately clear what selecting “Time left” would do (5), but after selecting it and returning to the clock it was clear

Application 2 – Class assistant

  • Prefers the simple ‘peek notification’ (1) since the alternative (3) might be hard to read on a small screen.
  • Showing the extended information (6) again might be unnecessary when the class has already started, maybe leave this screen out
  • Likes the confirmation of the saved recording (11)
  • If the smart watch has a camera, being able to quickly take a picture of the powerpoint would be useful

Application 3 – Shopping/cooking assistant

  • The ‘peek card’ shopping list is not good since you usually don’t check off a list from top to bottom in order while shopping
  • Likes the alternative where ticked off items move to the bottom of the list (5)
  • When ticking off the last item, expected to go straight to the first step of the recipe

Evaluation 5 – Computer science student, 22 years old, male

Application 1 – Pomodoro

  • Prefers analog clock but digital version might be useful as well
  • Proposes to make the duration sliders (3) vertical, so you don’t accidentally change it when trying to swipe to the next or previous screen
  • Was unsure if being able to change the duration of the sessions, during a session makes sense and what would happen if he tries it.

Application 2 – Class assistant

  • Likes the alternative notification (3) but would only want to remove the location
  • Would like the add the ending time of the class on the full card (2)
  • Would like to be able to quickly navigate to the shown location, either by tapping on the location or by adding a navigate button on a next screen
  • Mute/Unmute phone screen (7, 9) could be somewhat confusing
    • perhaps add a small icon indicating the current state of the phone
    • or add an extra screen showing the current state of the phone

Application 3 – Shopping/cooking assistant

  • The ‘peek card’ shopping list is not good since you usually don’t check off a list from top to bottom in order while shopping
  • Dislikes the alternative where ticked off items move to the bottom of the list (5), because an accidental double tap might result in two items being moved to the bottom, resulting in a lot of work and frustration to uncheck it again
  • When ticking off the last item, does not want it to return automatically to the clock, in case of an accidental input

Discussion

Application 1 – Pomodoro

Almost every test subject wanted to tap the clock to stop working. This is a good indication that it’s not very clear that you can swipe the clock. A solution could be to make the little circles at the bottom stand out more, or to actually implement the function that tapping the clock = stopping the work session. The only problem I could see with the latter would be accidental taps causing some frustration.

Most users prefer the analog clock, but wouldn’t mind an extra setting where they can choose between the two options. This is something which can be added very easily.

Application 2 – Class assistant

Some liked the simple notification (1), others the alternative (3). Since Android Wear apps always come with a side-app on the smartphone, perhaps we could let them choose which notification they prefer on the accompanying smartphone app. This could also easily allow users to choose which information to display on the expanded card (2).

The confusion with the mute/unmute buttons (7, 9), after discussing possible solutions with the test subjects, could most efficiently be resolved by rephrasing the command text to “Turn sound OFF” and “Turn sound ON”.

Some subjects wanted some extra indication when a recording was still going on. The proposed solutions were a small icon or a timer indicating the current length of the recording.

Application 3 – Shopping/cooking assistant

Nobody really liked the alternative of the ‘mini shopping list’ (6, 7). So this will definitely be removed or significantly altered.

The majority of the users liked the alternative of moving checked items to the bottom of the list (5), but evaluation 5 mentioned some good counterarguments regarding accidental taps. A solution would be to implement some sort of delay where you can’t quickly tap again after tapping. This could significantly reduce the possibility of accidental double taps.

Adding an extra button to quickly set a timer is something most users either proposed themselves or really liked the idea. This could easily be added by adding it on a new screen next to the “Recipe” button (2). Maybe a bit less clear would be to put a small button on a relevant instruction page (8).