Introduction
One critical aspect of designing successful products is their measurement and evaluation. This is the second principle of the user-centered design process: relying on empirical data. Its goal is the creation of highly usable products that are easy to use and error-free. As user experience professionals interested in improving the quality of our team's work, measurement is essential. Our job is to collect data that our team will use for analysis. There are different means to measure UX success, both quantitative and qualitative. Let us take a look.
Measuring at the beginning of product development
Evaluative research can occur at different stages in the product development life cycle. Assessing an existing product is quite common at the beginning of that life cycle. I evaluate a product based on standard usability guidelines and best practices. When I teach user experience, my curriculum often includes Jakob Nielsen's ten usability heuristics. I ask my students to evaluate an application based on his famous guidelines. I also like Tog's list of basic principles for interface design.[1] This type of evaluation, based on existing principles instead of speaking directly with users, is called a heuristic evaluation. It is considered a discount usability inspection method, intended to be a fast and efficient evaluation based on qualitative data.
Alternatively, we may also base our evaluation on criteria uniquely determined by the researcher for the task at hand. In the following example, I created a unique set of criteria and a scoring system to evaluate the Sony Music website, which adds a quantitative framework. I established five groups of questions for a total of 25 individual criteria. Each question had a possible score of -2 to +2, and the total range of scores was -50 to +50. A score of +25 I consider passing.
Additionally, my evaluation was through the lens of a user persona, who in this case was a 30-year-old man with two children, working in accounting, with a household income of approximately $75,000. He had specific goals of listening to and purchasing some music. He was also interested in connecting with others and learning more about artists. Some of the unique evaluative questions I asked included:
- Does the homepage provide evidence that users could complete their goals?
- Is essential content available where needed?
- Is the task flow efficient?
- Are keyword-based searches comprehensive and precise?
- Does the site help users recover from errors?
Each criterion in the review carried its own scoring guide. For the first question under Value, does the user interface provide evidence that user goals can be completed, the guide ran from -2, no evidence that any part of the goals can be completed, through -1, no evidence for key portions of the goals, to 1, evidence that users can likely complete their goals, and 2, evidence that they definitely can. The Sony Music site scored 2. The observations behind that score, one per persona goal, are below.
Sony Music heuristic review example
One page of 148 from the review. The full document covered 25 criteria with a custom scoring guide for each, scored from -2 to +2.
Evaluation criteria: Value. 1. Does the user interface provide evidence that user goals can be completed? Score: 2
| User goal | Result | Observation |
|---|---|---|
| Listen | Yes | There are a few play buttons that on click would allow one to listen to music. |
| Discover | Yes | The recommendations engine would suggest new artists based on the user's musical taste. More information is needed about how this tool works. One accesses this feature from the Recommendations tab. |
| Save | Yes | Users have the ability to add songs to playlists by clicking the Add to MyPlay List button. |
| Share | Yes | Clicking on the Send to a Friend button will allow one to share their music choices with others. |
| Purchase | Yes | The Buy It button suggests that one should be able to purchase music. |
The next criterion on the same page, is essential content available where needed, scored -1.
Measuring at the end of product development
Evaluative research may also occur later in the product development life cycle. The usability test is a standard research methodology. Typically, a group of users will evaluate the design team's work at this stage. This method occurs in a one-on-one format; sometimes participants form a group of two, a dyad, to conduct a usability test. Either way, a moderator guides the discussion and collects the participants' feedback. Afterward, their data are analyzed to make recommendations for improving the work.
Typically, the research team will provide a series of tasks for the participants to complete. Base these activities on ones your customers would perform in the real world. In these cases, your participants will interact with some form of prototype for your application. While your participants complete the tasks you have given them, they should also share their thoughts. This technique is called the think-aloud protocol for conducting usability testing. It asks your participants to speak about their thoughts while attempting to complete tasks.
Looking at the data: closed-ended questions
After your testing is complete, it is time to analyze the data. I recommend analyzing the data after each session. This is preferred because you will remember the conversations better, and the insights you gather from the last discussion will inform the next.
The first step is collecting closed-ended responses from items such as multiple-choice questions. In this case, structured data from an interview are similar to survey data. This activity resembles counting the number of answers to multiple-choice questions. I have also adapted my research practice to add a quantitative measure to most of my qualitative interview questions. I use a five-point Likert scale, with one being the lowest and five the highest. For example, I will ask my participants to rate how easy or difficult it was to complete a task, from one to five. Then I calculate the average or median scores. I can evaluate the change in that measurement over time as we improve our designs. I typically see a 2x improvement in those metrics.
Unstructured data: open-ended questions
It can be more challenging to analyze unstructured data from open-ended questions accurately. That has to do with the fact that each participant may have answered the questions differently. You also need to interpret their language to understand their intentions.
A common approach to looking at unstructured data is identifying all the information categories in your participants' responses. I make a simple list and note how frequently each category appears. You can organize this categorization by participant and across the overall interview study, including all participants.
For example, I may realize that a frequent category of information involves wayfinding, navigating to find desired information. I might count and document that my first participant mentioned this category ten times, and then note that it was mentioned 100 times across all of my participants. I also find representative quotes from the transcripts for each category. Finally, I watch how frequently certain categories of responses emerge over time. The goal is to reduce problematic areas with design improvements. In this way, I have found a quantitative metric for my unstructured data.
Surveys
This research methodology provides another excellent means to measure the user experience. I use surveys for both new and existing products, and I use them at the beginning or toward the end of the product development cycle. They are incredibly flexible. I consider surveys differently than user interviews and testing, which allow us to go more in depth on a topic. Surveys collect a broad range of data from large numbers of people. They also ask the same questions in the same order. They are highly structured, differing from user interviews in this regard.
One can use a variety of common quantitative data analysis calculations for a survey. These calculations are straightforward. I prefer to use the median, especially when an outlier or off-response may skew the data. Another beneficial statistical measurement is the range, the maximum value minus the minimum value. I use it to show the spread between the two extremes.
Conclusion
When it comes time to measure your product's user experience, it is good to sit down and brainstorm with your whole team: design, product, engineering, and any other stakeholders interested in your project. Typically, this yields a large number of product elements to evaluate. I like to put these product elements in an online spreadsheet I can easily share. Each team member can rank-order them and add their notes. With this in hand, the research team can determine which research methods to use to conduct their work. While we have discussed some of the most popular research methods, including heuristic evaluations, usability testing, and surveys, the good news is that our practice offers a variety of other techniques to capture the metrics that our team needs to build highly usable and successful products.