Expert reviews (also called heuristic reviews) can identify most of the usability issues that are likely to occur. In the hands of a competent designer, this type of review can be extremely effective. We have validated the quality of our expert reviews by conducting empirical usability tests on the pages we’ve reviewed. In general we find that careful and thorough reviews catch most of the problems with only a few additional problems being encountered in the usability test. Because expert reviews are less expensive than usability tests, we like to use them frequently.
Expert - or heuristic - reviews are an opportunity for the design team to obtain the perspectives of designers who have been outside the development process. There is probably no single more cost-effective way to improve the quality of a product.
Why Expert Reviews are Valuable
An outside perspective can be extremely valuable. When a software product is released, one of the most important questions is "will users find it intuitive"? Although the design team has struggled with this question throughout the entire project, they become so enmeshed in the process that they can no longer view it with naive eyes. They have been involved in many debates and compromises. In addition, the design team has confronted the realities of schedules, budgets, working with marketing and technical colleagues who may have different priorities and dealing with management expectations.
If we could predict the future, we could really reduce risk. Since we can't, the best we can do is to try to identify elements in the design which may be barriers to the users and impair the product's acceptance. The goal is to make the groups of successful users as large as possible and to have users feel that the software is a real asset - a tool that enhances their productivity.
Two tools that we have to help in this process are heuristic design reviews and usability tests. These two techniques complement each other. Usability tests show us how new users relate to the product on initial exposure. Unfortunately, usability tests are also expensive and time-consuming. And their yield can be relatively low compared to the effort expended. Usability testing is most useful at the beginning of a project, when we want to see if users "get" the basic design concept, and at the end, to identify any low-effort, high-yield refinements which are still possible.
A heuristic review is a consultative process. While there are a number of different techniques which can be used, the core process is to ask experienced designers, to work through the software, adopting as far as possible the viewpoint of a novice user. This is perhaps analogous to a teacher asking another educator to look at some instructional material to assess how well students will work with it.
Cognetics' approach is to conduct a heuristic review with 2-3 designers. This is a large enough group to provide a diversity of opinion yet small enough to be efficient. Before coming together, each designer looks at the software alone. The first look at a product is very valuable. It is important for the designer to make notes of initial reactions, misconceptions and blunders before he or she becomes too familiar with the software. Having walked through the software for initial reactions, the designer re-visits the software. He or she now looks at each screen looking for violations of the Cognetics heuristic guidelines. They note issues such as:
- What problems, if any, did I have here? Why was I confused and how might that confusion have been reduced or avoided.
- What elements might confuse novice users, even though I was not confused?
- What concepts and skills must the user have to understand the material presented at this point and the options available? Are there ways in which the presentation and options might be simplified.
- Are there demands on the user's recall (both conceptual and short-term memory) which could be reduced.
At the conclusion of this process, the designer should have developed a comprehensive set of notes, have reasonably well-developed ideas about design issues and possible solutions.
Now the design team comes together and performs a second walk-though of the software. As they go through it together, they raise and discuss the issues they noted. Some issues may be dismissed while new ones may emerge. Solutions are also suggested. At the conclusion of this process (usually ended by exhaustion) the combined set of notes is organized and a report created.
The results of the review are usually presented in a work session with the software design team. It is important that heuristic reviews not be seen as attacks on the design but as collegial input. It is not our role to make judgements about the design (like a movie critic) but to provide input that can help a software team achieve its goals.
The Expert Review Process
Expert reviews are often done as the beginning of a redesign project, as a way of correcting design or usability flaws before a formal usability test, or as part of a design engagement to solve a problem which cannot be resolved from within the current team.
The details of the schedule will vary, but typically they are structured in three steps:
- Cognetics reviews the software
The Cognetics designers receive, install and review the software on their own, to approximate the new users’ experience, and record their comments and observations. For complex software or task domains, an expert will need to be available to the review team to answer questions about the software and its typical uses.
- Cognetics assembles the notes from the review observations
Review observations are recorded and compiled with a brief summary describing the areas where the interface needs the most improvement.
- Cognetics works with the product team to solve usability problems found in the review
The Cognetics designers then meet with the product team to discuss the observations and work on design solutions to solve the problem. This session lasts from one to two days. A final day is spent, when necessary, working with the development or product lead to assemble the notes from the working session.
The goal of this review structure is to shift the emphasis from discovering problems to solving them, with the active involvement of the product development team. The review becomes part of the design process, rather than a "test" of the team's capabilities.