Complete and up-to-date documentation is essential for efficient data analysis in a large and complex collaboration like CMS. Good documentation reduces the time spent in problem solving for users and software developers. The scientists in our research environment do not necessarily have the interests or skills of professional technical writers. This results in inconsistencies in the documentation. To improve the quality, we have started a multidisciplinary project involving CMS user support and expertise in technical communication from the University of Turku, Finland. In this paper, we present possible approaches to study the usability of the documentation, for instance, usability tests conducted recently for the CMS software and computing user documentatio

Lassila-Perini, Kati

Salmi, Leena

CERN Document Server

Available on CMS information server CMS CR -2009/085The Compact Muon Solenoid ExperimentMailing address: CMS CERN, CH-1211 GENEVA 23, SwitzerlandConference Report08 May 2009Improving collaborative documentation in CMSKati Lassila-Perini, Helsinki Institute of Physics, FinlandLeena Salmi, Department of French Translation Studies, University of Turku, FinlandAbstractComplete and up-to-date documentation is essential for efficient data analysis in a large and complexcollaboration like CMS. Good documentation reduces the time spent in problem solving for users andsoftware developers. The scientists in our research environment do not necessarily have the interestsor skills of professional technical writers. This results in inconsistencies in the documentation. Toimprove the quality, we have started a multidisciplinary project involving CMS user support and ex-pertise in technical communication from the University of Turku, Finland. In this paper, we presentpossible approaches to study the usability of the documentation, for instance, usability tests conductedrecently for the CMS software and computing user documentationPresented at CHEP09,21 - 27 March, 2009,Prague,Czech Republic,15/05/2009      Improving collaborative documentation in CMS Kati Lassila-Perini1 and Leena Salmi21Helsinki Institute of Physics, P.O.Box 64, FI-00014 University of Helsinki, Finland  2Department of French Translation Studies, University of Turku, FI-20014 University of Turku, Finland E-mail: k.lassila-perini@cern.ch Abstract. Complete and up-to-date documentation is essential for efficient data analysis in a large and complex collaboration like CMS. Good documentation reduces the time spent in problem solving for users and software developers. The scientists in our research environment do not necessarily have the interests or skills of professional technical writers. This results in inconsistencies in the documentation. To improve the quality, we have started a multidisciplinary project involving CMS user support and expertise in technical communication from the University of Turku, Finland. In this paper, we present possible approaches to study the usability of the documentation, for instance, usability tests conducted recently for the CMS software and computing user documentation. 1.  Context: CMS analysis software documentation The CMS software documentation suite consists of a Workbook (getting started instructions) and an Offline Software Guide (with more detailed information), implemented online on the CERN TWiki [1] platform. These documents cover all software residing in the common CMS software repository (CMSSW), and the instructions on how to use this software in the CMS computing environment. The document suite is managed by the CMS User Support and the contents are authored by physicists developing the code. Authors themselves are often also the target audience for different areas of the documentation suite. The authors have a high level of expertise in the areas which they cover but they may lack experience in technical writing.  Therefore, part of the documentation is either incomplete or too detailed from the user’s point of view. In addition, all the existing documentation has not yet found its way to the suite and all areas are not fully covered. 2.  CMS software documentation review: goals and constraints The CMS User Support has started a review to improve the usability of the documentation suite. According to the ISO standard 9241-11 [2], the usability of any computer system means the extent to which users are able to use it, in a given context, to achieve specified goals effectively and efficiently. The objective of this review is to achieve a predictable and consistent documentation structure and to complete the missing areas for the comprehensiveness of the suite. Simplifying, a documentation suite should provide easy-to-find answers to users’ basic questions:  what, why, how, where, when?       The review is constrained by the lack of experience in technical writing of many contributors. A completed documentation is not required in the release procedure of the software and therefore we rely on only voluntary contributions. Furthermore, manpower resources are limited. In the following, the review process is described in detail and the results of the first step are summarized. 3.  How to achieve these goals? In order to improve the documentation, the first step was to find out in detail what was problematic in its usability. Well-proven methods exist to evaluate and improve usability. Nielsen’s book [3] is a basic reference on usability and provides good guidelines for evaluating it. Also Ivory and Hearst [4] have made a comprehensive survey and a classification of the different methods described in scientific literature. They divide usability evaluation methods into five classes: testing, inspection, inquiry, analytical modeling, and simulation. Methods within the first three classes are used for formative and summative purposes – with them, the evaluators get data from either users (interacting with the user interface or commenting on it) or by assessing the user interface themselves. The last two classes of methods are used to predict usability problems and they have an engineering approach rather than the goal of evaluating existing user interfaces. The methods listed in [4] apply to user interfaces on the computer, both software applications and web sites. As the authors point out, very often the role of a web user interface is to provide information rather than to help users complete a task, and this is the case with the CMS software documentation.  In the following, we describe what methods were used in the CMS software review process and in what way. 3.1.  CMS software documentation review process The review process started with usability test sessions (see 3.2.1) to address the accessibility of the information on a general level and to pinpoint problems. Each group providing software to the common CMSSW repository then undertook a review consisting of • heuristic evaluation (see 3.2.2) to set the group-specific goals • intense, short working period where group members need to get involved • final review (see 3.2.3) • group-specific usability test. The review started with the Physics Object Groups (POG) whose deliverable is the software description of physics objects (electrons, muons, jets, etc) in the software repository. It was continued by some key groups involved in the software development (Physics Analysis Tools, Event Generation and Simulation), and with the Physics Analysis Groups (PAG). 3.2.  Methods Documentation has been the object of usability evaluations less often than software applications, and such evaluations have not been used in the context of the documentation for large high energy physics collaborations – the innovation in this review is to integrate them to the software development work flow in a collaborative manner.  Among the methods described in [3] and [4], we have chosen usability testing and heuristic review of the documentation. In the following, we describe why the methods were chosen and how they were used. 3.2.1.  Usability tests. The aim of usability tests is to gain information on the actual use by real users of the user interface being tested. This is done by asking test users to do typical tasks with the user interface, and to record the test sessions. To get information on what the users find problematic, they are asked to think aloud while they are working. It is recommended that the time spent in a usability       test should not exceed one hour, as this type of work with thinking aloud is rather tiresome [5]. In an analysis phase, the recordings are reviewed and transcribed and the usability problems are listed.  According to Nielsen [6], five test users would be enough to find even 85% of the usability problems, but as this statement has been criticized, among others, by Woolrych and Cockton [7], we decided to have eight users participate in the testing.  In the CMS software documentation review, we first arranged eight test sessions where test users were given typical tasks, such as: • “You are planning to start analysis, where would you go to look for information?” • “You will have e-/μ/jet in your analysis, find what the e-/μ/jet data contain” • “Find a useful example code to access data” There were five questions altogether, but if it started to look that it would take for more than an hour for the user to complete the task, one of the tasks was left out.   Another set of usability tests is being planned at the end of the review process, after the corrections suggested by the initial usability testing and the heuristic evaluation have been implemented. 3.2.2.  Heuristic evaluations. Ivory and Hearst [4, p 487] define usability inspection as a method whereby “an evaluator examines the usability aspects of a user interface design with respect to its conformance to a set of guidelines”. Such an inspection method is heuristic evaluation, where evaluators check the compliance of the documentation with recognized usability principles. According to Nielsen [3], there should be about five evaluators that inspect the user interface first on their own and then their findings are put together. Nielsen [8] recommends combining heuristic evaluation with usability testing, as both methods have been shown to find different sets of usability problems.  In this review, we applied the heuristic evaluation methodology to review the user interface with the developers, to discuss the usability problems and to find solutions to fixing them. We organized an evaluation session for each group separately with the CMS User Support and with the person nominated to manage the review process of the group. These evaluation sessions are currently ongoing for a part of the teams. In the evaluation session, the documentation is gone through by comparing each part to a predefined check-list (see 3.2.3). The outcome of each review should be a set of recommendations for authors on how to improve the documentation and a set of goals which can realistically be reached within a short working period. Organizing evaluation sessions is a time-consuming procedure but it is significantly more efficient than passing the information by e-mails or by written instructions. It also gives an occasion for informal feedback and reflection on the documentation needs of the different groups.   3.2.3.  Use of a check list. Vesa Purho [9] has developed a check-list of heuristics suitable specifically for evaluating the usability of documentation. The list contains 10 recommended heuristics. The most relevant parts to CMSSW documentation review are  • match between documentation and the real world • match between documentation and the product o These two statements may contradict: we have recommended – in case of documenting the algorithms or C++ classes – to use the class name (match the product) as a link and a short one-line description of its main use (match the real world) after the link. • support for different users o Our documentation addresses the needs of  users who want to know how to use the algorithm and of developers who need to know who the algorithm works: we have recommended to clearly separate these two parts by titles “Usage of algorithm” and “Implementation details”.  • effective information design       o The authors were reminded that long paragraphs should be avoided in pages directing to further information as most readers only read the links – any statements in plain text paragraphs will very likely be ignored. In addition, ISO provides check lists to review the user documentation [10]. It addresses different areas, among which the contents of the documentation, and gives a detailed set of questions to be answered to verify that all documentation areas are fully covered. These areas are • general information • overview of the application • overview of the documentation • task descriptions • parameters sets • user interface elements • application functions • messages • terms • concepts • questions and problems • examples • captions.  The managers of the group-specific reviews have been encouraged to refer to this list. 4.  Results from the usability tests The analysis of the data from the usability tests was started by transcribing the contents of the session. The transcription included listing the pages that the user visited and noting the essence of their comments. In order to compare the “path” the user was taking to complete the task to the “optimal” path the developer of the documentation had in mind, some of the sessions were visualized as flow charts containing both the optimal and the user’s actual pathway (see Figure 1), using a visualization method developed in Salmi [11]. Figure 1 shows a path of one test user looking for information for the first task: “You are planning to start analysis, where would you go to look for information?”.        Figure 1. An example path of one test user and the optimal path to the requested information analysed by a method developed in [11].        The usability tests clearly indicated problem areas in the accessibility. In general, it took longer than expected for most of the test users to arrive to the requested information. They were often distracted to non-related pages and they spent time scrolling through pages trying to judge if they could contain information they were looking for. The main observations are detailed in the following. 4.1.  Access patterns  The access pattern was common to most test users: they looked for the entry point to the software documentation from the Physics Analysis Group (PAG) or in the Physics Object Group (POG) home pages. However, these pages do not and should not contain information useful to get started in writing the analysis code but they should link to the appropriate documentation pages – this was, however, very rarely the case. The Offline SW Guide includes pages for each PAG and POG group which are meant to contain all documentation on group-specific software and instructions on how to use it but this page was often not even linked from the groups’ home pages and in most cases it was incomplete. This point is being emphasised in the review of the group documentation, links to general guidelines and to the documentation are being added to each group’s home page and the software documentation is being completed in the Offline SW Guide PAG pages which were often the less completed part in the CMSSW documentation suite. 4.2.  Accessibility All test users found the Workbook, but the Offline SW Guide was generally not found. It was linked from the Workbook in the right-hand side of the page but the test users never used this link. When they were shown the Offline SW Guide page, they were pleased to find a navigation bar giving access to different documentation areas and quick links to most frequently needed pages such as instructions on how to access to data and data formats. This navigation bar has now been made common to the Workbook and the Offline SW Guide and it appears also on the Workbook entry page to ensure that Offline SW Guide contents are easily found. 4.3.  Keywords The test users looked for keywords such as “CMSSW”, “Getting started”, “Examples”, “How to”, “Tutorials” etc. We observed that they did not find some documentation pages containing examples because the title of the link to the material did not clearly indicate so. When the test users found keywords mentioned above they usually explicitly stated that this material looks what they are looking for. A page containing a chronological list of computing and software tutorial sessions organized at CERN and the links to the material shown in this session attracted most of the test users. It is linked from the main Physics, Computing, Offline and User Support pages under title “Tutorials” and from the Workbook right hand-side bar. Curiously, this link was seen and followed whereas the link to the Offline SW Guide just below was not. However, the tutorial page in its current format does not correspond to the needs of the users. The test users did find the information, but they often ended up to the slides shown in the tutorial sessions while the updated, complete information would have been available in the Workbook. The tutorial page will be restructured following the users’ tasks and needs rather than a chronological order and the direct links to the Workbook tutorials will be provided immediately on this page. 4.4.  Navigation habits The test users had very different navigation habits. As an example, all Workbook pages have a left-bar with a list of contents of the Workbook – some test users never used it while others often navigated using it. Another example is the Workbook main page which is given in a table of contents format – some test users read it through from the start to the point where they found the information they were looking for while others quickly scrolled the page without detailed reading of the titles.        5.  Conclusions Small things matter! Most problems encountered by the test users were not connected to information not being provided but to information not being found. Small improvements – such as bringing the reader to the appropriate documentation by adding links where they would have expected them – can make a big difference for the user. Usability tests are an excellent tool for pinpointing the problems.  Some immediate improvements have already been made: link titles have been modified to better correspond the users’ tasks and expectations, authors have been provided template pages and a navigation bar giving immediate access to the comprehensive documentation suite has been added. Individual evaluation sessions for each group are time-consuming but essential for defining clear group-specific goals. Documentation reviews are ongoing and will be completed by group-specific usability tests.  References [1] http://twiki.org [2] ISO standard ISO/IEC 9241-11:1998 (E) [3] Nielsen J 1993 Usability Engineering. London: Academic Press [4] Ivory M and Hearst M 2001 The state of the art in automating usability evaluation of user interfaces ACM Computing Surveys 33 470-516 [5] Sinkkonen I, Kuoppala H, Parkkinen J and Vastamäki R 2002 Käytettävyyden psykologia [The psychology of utilisability] Helsinki: Edita Oyj/IT Press [6] Nielsen J 1994 Estimating the number of subjects needed for a thinking aloud test International Journal of Human-Computer Studies 41 385-97 Nielsen J 2000 Why you only need to test with 5 users Jakob Nielsen’s Alertbox, March 19, available at http://www.useit.com/alertbox/20000319.html  [7] Woolrych A and Cockton G 2001 Why and when five test users aren’t enough People and Computers XV – Interaction without frontiers. Joint Proceedings of HCI 2001 and IHM 2001, Volume II ed J Vanderdockt, A Blandford and A Derycke (Toulouse: Cépaduès-Editions.) pp 105-8 [8] Nielsen J 2005 Characteristics of usability problems found by heuristic evaluation. Paper available at http://www.useit.com/papers/heuristic/usability_problems.html [9] Purho V 2000 Heuristic inspections for documentation – 10 recommended documentation heuristics Usability Interface 6. Available at http://www.stcsig.org/usability/newsletter/0004-docsheuristics.html [10] Annex B of the ISO Standard ISO/IEC 18019:2004(E), copyright waived by ISO [11] Salmi L 2003 Documents multilingues pour logiciels et utilisabilité Ph.D. thesis, Annales Universitatis Turkuensis B. 269, Turku: University of Turku 75-83  

Improving collaborative documentation in CMS

Abstract

Similar works

Full text

Available Versions

CERN Document Server