6. User Evaluation
We performed a user study in order to evaluate the effectiveness of MMNail document visualization compared to viewing a document in a PDF viewer on a small size viewing area. Our hypothesis is that MMNail viewing provides users with document comprehension superior to that of PDF viewing.
6.1. Evaluation Setup
A total of 16 users participated in our user study. Most users had technical backgrounds and even though all of them were very familiar with use of computers only six participants regularly used hand held devices for tasks other than making phone calls.
One of the two documents shown to the users was a technical paper on a photo browsing system and the other one was a magazine article on San Francisco's Angel Island. The first paper is 8 pages long and the second article is 4 pages long. Content included text, graphics, and pictures.
In our user study, a laptop computer is used to show both an MMNail viewer and PDF viewer to the users. A meeting recorder system [27] is used to capture the laptop screen and conversations with the users. A laptop is used instead of a handheld device because of its adequate speed and ease of screen capture. Only a limited viewing area (320×240 pixels) is used on the laptop computer and users' interactions with PDF viewer interface were limited to zoom-in, zoom-out, and pan, similar to typical interactions on handheld devices. For the MMNail visualization the interface described in Section 5 is used on the same laptop using the same viewing area.
Each user viewed one document in MMNail format and the other document, which has different content than the first one, in PDF format. We employed a rotation design by alternating the order of the viewing environments in order to keep the dependency on viewing order as small as possible. Users are given limited time to view and interact with each of these documents: 2 minutes 40 seconds for MMNail and PDF versions of the technical document and 2 minutes for both versions of the magazine article.
6.2. Task and Evaluation Measures
The users' overall task is to understand each document's content and to recognize visually the parts of the document after a limited-time browsing. In order to evaluate the user's comprehension of the content, after document browsing users were given a quiz that contained the following two parts:
- Textual content quiz: Users are given a list of 20 topics, out of which 10 topics are covered in the document and 10 topics are not covered in the document. Topics included phrases such as "group annotation" and "mobile blogging" for the technical paper on photo editing. Users are asked to mark the topics that they think appeared in the document.
- Visual content quiz: Users are given a list of 10 visual document elements, e.g., page thumbnails or figures, out of which 5 elements are extracted from the document and 5 do not belong to the document. These visualizations are presented in pairs to users, who are asked to pick which one of the visual representations is included in the document.
It is important to note that the document topics were extracted by a professional librarian who saw only the paper copies of the documents and did not know how the MMNail algorithm works. This removes the possibility of any bias in the preparation of quiz questions.
6.3. Comprehension Quiz Results
Table 3. Summary of quiz scores when users view MMNails vs. PDF files.
| Viewing method/Document type | Textual content | Visual content | ||
| Score average | Score σ | Score average | Score σ | |
| MMNail/Technical Document | 54 | 18 | 41 | 15 | PDF Viewer/Technical Document | 39 | 15 | 33 | 23 |
| MMNail/Magazine Article | 71 | 21 | 45 | 7 |
| PDF Viewer/Magazine Article | 58 | 15 | 48 | 9 |
| MMNail/Average of both documents | 62 | 19 | 43 | 11 |
| PDF Viewer/Average of both documents | 49 | 15 | 41 | 16 |
Table 3 presents the results of quiz scoring for both MMNail and PDF viewing. Quiz scoring is done such that a wrong answer cancels a right answer. It can be seen from the table that users obtained an average score of 62% for the MMNail viewing vs. 49% for PDF browsing of documents when they were asked questions about the textual content. Quiz scores were 43% for MMNail viewing and 41% for PDF viewing of documents when users are quizzed about the visual content of the documents. As can be seen from these results, viewing MMNail documents, users performed slightly better in visual quiz and significantly better in textual quiz compared to viewing PDF documents. Statistical analysis was performed using a paired sample t-test showing statistically significant difference between the textual content comprehension using MMNails and PDF viewer with a p-value of p = 0.05. For visual content comprehension no statistically significant difference can be reported.
6.4. Questionnaire Results
In addition to the quiz we gave each participant a questionnaire in order to receive feedback on the overall viewing experience of MMNails. The questionnaire results are presented in Table 4. In average, users rated MMNail viewing experience as a pleasant experience and indicated that MMNail viewing is better than PDF viewing. They also found the audio channel and automatic navigation useful. Variance of their scoring was highest in the case of usefulness of audio, which may be an indication of benefiting from audio channel being more of a personal choice.
Table 4. User's scoring of MMNail viewing experience on a scale of 1 to 10.
| Questionnaire topics | Score average | Score σ |
| Overall viewing experience [1: unpleasant, 10: pleasant] | 8.45 | 1.43 |
| MMNail compared to PDF viewing [1: MMNail worse than PDF, 10:MMNail much better than PDF] | 8.24 | 1.86 |
| Usefulness of audio synthesis [1:not useful, 10:very useful] | 7.88 | 2.69 |
| Usefulness of automatic visual navigation [1:not useful, 10:very useful] | 8.47 | 2.07 |
Users' Comments
In an exit questionnaire users were asked to comment on their MMNail viewing experiences. Generally users commented that document skimming with MMNails was less stressful compared to manually skimming of PDF files. Because important document parts were already identified in the MMNail, users felt that they were viewing the parts of the document that were relevant. Users also thought audio was good for capturing the browsing person's attention, and it was easy to remember a topic when it was both seen and heard. As a shortcoming, users stated that sometimes they felt that they were not in control of the viewing experience or lost in the document. They suggested user interface components that would overcome these limitations as summarized in the next section.
