About Virta

This fall, as part of our Usability Evaluation Methods course at Tampere University, we conducted usability tests in collaboration with Virta in a group of 5. Virta is a type of bulletin board that contains different topics occurring in Finland and its main goal is to let Finns abroad have a say on important matters or topics that interest them. The main functions of Virta are:

  1. Search among the available hearings

  2. Read hearings

  3. Comment on hearings and reply to other users’ comments.

Background

As part of our course, we first carried out a heuristic evaluation based on Don Norman’s 10 Usability Heuristics and assessed usability problems based on Jakob Nielsen’s severity rating scale ranging from 0 (Not a problem) to 4 (Usability catastrophe).

In total, we discovered 42 usability issues in the following categories: general, registration, login, search, topic suggestion, commenting, and hearing creation.

Goals of the Project

Although 'hearing creation' was by far the most error-prone section of the website, we decided not to focus on it because a typical user of the service cannot create a hearing on the website; only admins, who constitute another target group, have access to it.

Additionally, since we did not want participants to enter private information, we also skipped the registration section of the service.

Consequently, our set goal before conducting usability tests with potential users was:

“To identify usability problems on the website, particularly in the search and commenting functions.“

Test Procedure

As the usability tests were part of our university course, we had the privilege of conducting them in one of the university's usability labs. The lab comprises two rooms divided by a one-sided window. The test is conducted in one room, while the other room serves as an observation area for observers to watch the test and address technical issues.

During the tests, the participant's screen was recorded for subsequent analysis, and their faces and voices were also recorded to aid in interpreting body language and auditory cues while performing tasks. Only one usability test was conducted online due to my contracting COVID-19 at the scheduled time.

The test began with the moderator introducing themselves and providing a brief background on the service and the purpose of the usability test. Once the test procedure was explained, participants were asked to sign the consent form and fill out a background questionnaire.

The concept of 'thinking aloud' was then introduced and taught to them; this self-report method was employed to encourage participants to share their thoughts while completing tasks to gather valuable data. Next, participants engaged in a practice task to familiarize themselves with the 'thinking aloud' process.

Subsequently, the test tasks were presented one at a time. We created a test account for participants and initiated the test with the user logged in, providing login credentials on paper in case of any logouts during the test. Printed copies of the tasks were also provided, and participants, upon completing each task, informed the moderator to receive the next one. For the online test, we prepared a Mural canvas to provide the participant with the necessary information and test tasks.

After finishing all tasks, participants were directed to complete a post-test questionnaire, specifically the System Usability Scale (SUS) questionnaire, to rate their experience with the service. Finally, participants were asked a couple of open-ended questions to gather qualitative and in-depth data about their experience using the service.

Roles

The team designated roles during the usability tests, with distinct responsibilities for each member:

Moderator

For each test, one of our teammates assumed the role of the moderator, responsible for facilitating the usability test and providing assistance to the participant when needed, without leading them.

Observers

During each test, at least two team members served as observers, taking notes and addressing technical issues.

Test Materials

The materials used in the test included tasks, a script, a consent form, a background questionnaire, a System Usability Scale (SUS) questionnaire, and an interview frame.

Test Tasks

With a specific focus on the searching and commenting functions and aiming to maintain a realistic task order, 13 tasks were created. These tasks commenced with a practice task allowing participants time to become familiarized with thinking aloud. Following this, a zero task was introduced to relax participants and enable them to explore the service independently. Subsequent tasks were designed to reveal usability problems on the website. The list of all the tasks is as follows:

Practice Task: Open Google Maps and find out how you can go to the Hervanta campus from the City Center campus by car.

Task 0: Find out where you can learn about Virta.

Task 1: On the same page, read the first 2 paragraphs under the “What is Virta?” heading.

Questions for Task 1:

1. What do you think this service is for?

2. What do you think is a hearing?

Task 2: Find out how many hearings are currently available.

Task 3: Find hearings related to culture.

Task 4: Find the oldest hearing with content about a usability test.

Task 5: Rearrange the hearings again so that the hearings closing last appear first in order.

Task 6: Find and open a hearing related to nature.

Task 7: In that hearing you opened about nature, reply to a user’s comment saying “Hello world!”

Task 8: In the same hearing, reply “hei hei” to a user.

Task 9: Then add a new comment, saying “terve” with an image of your choice.

Task 10: Edit the same comment you just added.

Task 11: Then find and remove the "hei hei" comment you replied to a user.

The Script

The script was written aiming to achieve three main goals:

  1. Smoothly introducing the test procedure and guiding participants from one step to another.

  2. Ensuring participants understood that usability tests were evaluating the service, not them, and encountering problems was helpful for insights.

  3. Clearly elaborating on "thinking aloud" to help participants understand how they could contribute more information about their thought process.

Consent Form

To record participants' video, audio, and screen for further analysis, a consent form was prepared and participants were asked to sign it before the test started.

Background Questionnaire

Before participants performed the tasks, they filled out a background questionnaire, collecting information such as age, gender, native language, occupation, and whether they had used the system before.

System Usability Scale (SUS) Questionnaire

After completing all test tasks, participants were asked to fill out a System Usability Scale questionnaire, rating 10 statements on a scale from 1 (Strongly Disagree) to 5 (Strongly Agree) based on their experience of using the system. The questions are as follows:

  1. I think that I would like to use the website frequently.

  2. I found the service to be simple.

  3. I thought the service was easy to use.

  4. I think that I could use the service without the support of a technical person.

  5. I found the various functions in the service were well integrated.

  6. I thought there was a lot of consistency in the service.

  7. I would imagine that most people would learn to use the service very quickly.

  8. I found the service very intuitive.

  9. I felt very confident using the service.

  10. I could use the service without having to learn anything new.

Interview

Following the SUS questionnaire, a semi-structured interview was conducted to gather more detailed data about participants' experiences. This also allowed for clarification in case participants had said anything unclear while thinking aloud. The questions were as follows:

Theme 1: General queries

  • How would you describe your overall experience with the service?

  • Were there any tasks during the test that stood out to you? Why?

  • If you could change anything about it, what would you change?

Theme 2: Search functions

  • How did you find the search functions in this service?

  • What, if anything, caused you difficulties during the tasks related to searching hearings?

  • If you could change anything about the search functions, what would you change?

Theme 3: The comment section

  • How did you find the comment section in this service?

  • What, if anything, caused you difficulties during the tasks related to the comment section functions?

  • If you could change anything about the comment section, what would you change?

Results

We recruited four participants, and their background information is shown below.

List of the Usability Problems Found

Here is a list of all the common and significant usability problems discovered during the usability tests, indicating whether each participant encountered each problem or not.

  1. What is meant by "hearing" is not clear even after reading the about page.

  2. The placeholder text in the search bar is in Finnish, even though the user has selected English in the language settings.

  3. The label list in the "Limit with label" function has words both in Finnish and English.

  4. The label list is not in any specific order that would help the user to find what they are looking for (e.g. alphabetical order).

  5. Sort selection changes to default value (newest first) after a user chooses a label from the "Limit with a label" function.

  6. Closing last meaning is unclear for the user, e.g. closing can be mistaken with physically closing a hearing window.

  7. Typing a keyword relevant to a hearing in the search bar leads to no results.

  8. The date formatting system is difficult to distinguish if it's the US or the EU system and does not indicate the year.

  9. The comment section always closes after a comment is submitted.

  10. Comment section closing is inconsistent, e.g. closes when submitting a comment but remains open after editing a comment.

  11. The selected language (English) is not reflected for deleted comments which can be mistaken for an active comment or cause confusion.

  12. If all the hearing's sections are closed there is no visible feedback about the system’s status after a user clicks on "Write a comment".

  13. There is no efficient way to allow users to find and distinguish the comments they have posted without manually searching for their comments.

Success Rate and Task Time

To measure effectiveness, we calculated the average success rate of the 12 tasks we gave each participant. Additionally, we calculated the average task time to measure the efficiency of the service.

The average completion rate for all tasks combined is 93.75%, and each task, on average, took 1 minute and 10 seconds to be completed. This significantly high completion rate and short task times indicate that the service is effective and efficient; an average user can achieve their goal using various functions of the service in a short amount of time, despite experiencing some cognitive load.

SUS Score

Based on responses to the System Usability Scale questionnaire, we calculated each participant’s SUS score. Generally, the average SUS score is 68, making any score above 68 considered above average. To interpret the scores, we also employed Bangor’s adjective rating scale, which spans seven levels from 'worst imaginable' to 'best imaginable.'

The data in the table reveals that 75% of participants achieved SUS scores above 68, with one participant scoring 92.5, interpreted as 'best imaginable.' This suggests that most participants found the product usable. Additionally, the average SUS score for all participants is 75.62, exceeding the average of 68 by 7.62 points. This indicates that the product is considered more usable than the average product in the market.