This plan will be constantly updated and expanded in line with the planned software development activities related to the KRSM component of the TENCompetence infrastructure.This report provides a detailed plan for experimentation and evaluation of the KRSM prototype.The work on this publication has been sponsored by the TENCompetence Integrated Project that is funded by the European Commission's 6th Framework Programme, priority IST/Technology Enhanced Learning. Contract 027087 [http://www.tencompetence.org

Grigorov, Alexander

Krassen, Stefanov

Luccini, Marco

DSpace at Open Universiteit Nederland

 
Building the European Network 
For Lifelong Competence Development 
 
TENCompetence IST-2005-027087 
 
 
 
Project Deliverable Report 
 
Milestone M5.8 KRSM first cycle prototype evaluation plan 
 
Work Package Work Package 5 
Task Task 5.3 
Date of delivery Contractual: 01-1-2007 Actual: 22-1-2007 
 
Code name  Version: 1.0 Draft  Final  
 
Type of deliverable Report 
Security 
(distribution level) 
Public 
Contributors Alexander Grigorov (SU), Krassen Stefanov (SU), Marco Luccini 
(GILABS) 
Authors (Partner) Alexander Grigorov (SU) 
Contact Person Krassen Stefanov (SU) 
WP/Task responsible Marco Luccini (GILABS) 
EC Project Officer Hans-Juergen Westhoff 
Abstract  
(for dissemination) 
This report provides a detailed plan for experimentation and 
evaluation of the KRSM prototype.  
After the execution of the planned experimentation and evaluation 
activities, an Evaluation report will be delivered at month 16.  
This plan will be constantly updated and expanded in line with the 
planned software development activities related to the KRSM 
component of the TENCompetence infrastructure.  
Keywords List Evaluation, testing, experimentation, validation, software quality, 
quality assurance, usability.  
 
TENCompetence Project Coordination at: Open University of the Netherlands 
Valkenburgerweg 177, 6419 AT Heerlen, The Netherlands 
Tel: +31 45 5762624 – Fax: +31 45 5762800 
 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 2 / 47 
 
Revision History 
 
Date Version Description Author 
29-09-2006 0.1 First draft version of the report Alexander Grigorov 
03-01-2007 0.5 Complete version with some specific data still 
missing 
Alexander Grigorov, Krassen 
Stevanov 
22-01-2006 0.9 Almost final version Alexander Grigorov, Krassen 
Stevanov, Marco Luccini 
25-01-2006 1.0 Incorporation of feedback  Alexander Grigorov, Krassen 
Stevanov, Marco Luccini 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 3 / 47 
 
Table of Contents 
 
TABLE OF CONTENTS ..........................................................................................................................3 
1 INTRODUCTION............................................................................................................................4 
2 METHODOLOGY.............................................................................................................................4 
2.1 Objectives and scope .................................................................................. 4 
2.2 Evaluation Criteria ....................................................................................... 5 
2.3 Which quality attributes will be tested ................................................ 6 
2.4 Types of Testing ........................................................................................... 9 
3 EVALUATION OF THE KRSM SYSTEM .............................................................................11 
3.1 Objectives and scope ................................................................................ 11 
3.2 Test approach .............................................................................................. 12 
3.3 Evaluation Schedule .................................................................................. 12 
3.4 Roles and Responsibilities....................................................................... 13 
3.5 Test Environment ....................................................................................... 14 
3.6 Evaluation of the Quantity and Complexity of the System........ 14 
3.7 Evaluation of the Quality of the Software ........................................ 15 
3.8 Design of Test Clusters, Conditions and Cases .............................. 15 
3.9 Test Execution ............................................................................................. 17 
3.10 Evaluation Report ................................................................................... 18 
4 NEXT STEPS ..................................................................................................................................18 
5 REFERENCES.................................................................................................................................19 
6 APPENDICES.................................................................................................................................20 
6.1 General Questionnaire on the Quality of the Software ............... 20 
6.2 Coding Quality Questionnaire of the KRSM System ..................... 22 
6.3 Templates for Test Clusters, Conditions and Cases ..................... 24 
6.4 KRSM Test Cases ........................................................................................ 28 
6.5 KRSM Functionality Questionnaire....................................................... 46 
 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 4 / 47 
 
1 Introduction 
 
This document presents the experimentation and evaluation plan of 
the knowledge resource sharing and management components. This 
plan will first guide the gap analysis to be performed on the first cycle 
KRSM releases. After that, the evaluation of the second cycle release 
will be performed between month 24 and month 28. The evaluation 
plan will be updated and upgraded according to the emerging needs 
that may occur during the project period lifespan. 
 
The outcome of these evaluations will be used for the improvement of 
the system and will be given as input to task 2 of WP5. 
 
2 Methodology 
2.1 Objectives and scope 
 
The evaluation of a research project always raises many questions and 
challenges. 
 
A first set of questions is related to the objective of the evaluation. Is 
the evaluation conducted to guarantee that the resources have been 
properly utilized for what they were intended, or is the objective of the 
evaluation to provide the participants an assessment and some 
feedback that will help them to better pilot the project and, in 
particular, maximize the generation of value through this project?  
 
A second set of questions is related to the scope of the evaluation. Are 
we interested in assessing the process of advancement of the project 
or in evaluating the quality of the results that are generated by this 
project? Are we interested in evaluating the technical system (the 
demonstrator) that is being designed, or in the approach that this 
system is expected to validate? 
 
An additional set of questions has to do with carrying out the 
evaluation: What amount of resources should be dedicated to the 
evaluation of the project? How can we evaluate the effort, and, 
especially, decide how the evaluation resources are to be allocated? 
How should we direct the effort (prioritization)? How do we deal with 
all the risks associated with the evaluation and, in particular, the 
resistance of people and organizations to participate in an activity that 
consumes their time, and may threaten their position? 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 5 / 47 
 
 
The final set of questions is related to the analysis of the evaluation 
results and the use of the evaluation. How do we get the most out of 
this evaluation, identify the most significant results and learn from 
them? 
 
Answering all these questions is difficult, and is well beyond the scope 
of this document. 
 
Indeed, if the main focus of a research project should be the 
maximization of the effectiveness of the evaluation effort in the 
perspective of the value of the generated knowledge (value for the end 
user; novelty of the solution; capability to exploit this knowledge), a 
project very rarely provides the time to evaluate all the potential 
impacts on the society of the knowledge that has been created. 
 
2.2 Evaluation Criteria 
 
The KRSM system should comply with the following software scope 
qualitative criteria: 
• Relevance: How relevant is the software for the further 
development of the domain? 
• Significance: How important is the problem addressed by the 
software for the domain? Does the software have a community 
of users? 
• Originality: Are the problems and approaches new? Is this a 
novel combination of existing techniques? 
 
The software quality of KRSM components should be evaluated using 
the ISO 9126 [1] quality attributes and guidelines as described in 2.3. 
 
The software coding quality should meet also a number of quality 
assurance criteria (code readable, code commented, code structured, 
etc.) as formulated in the TENCompetence Handbook. 
 
The evaluation should also consider: 
• the quantity and complexity of the system, identifying clearly 
new developments and re-usability of existing components;  
• the impact of the system (what is the added value, is it 
downloaded from the CVS from other users, what is their 
opinion, etc.); 
• the extent the KRSM components meet the functional 
requirements of the system. 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 6 / 47 
 
 
2.3 Which quality attributes will be tested 
 
ISO 9126 [1] gives guidelines and describes the quality attributes that 
could be used for the evaluation of a software product. The ISO 9126 
model defines six product characteristics (see Figure 1): 
• functionality; 
• reliability; 
• usability; 
• efficiency; 
• maintainability; 
• portability. 
 
Figure 1: ISO 9126:1991 
These six characteristics (attributes) are further subdivided into a 
number of sub-characteristics. Table 1 presents the quality attributes 
and their description. 
 
Quality attributes Description: the capability of the software product to… 
Functionality provide functions which meet stated and implied needs when the 
software is used under specified conditions. 
Suitability provide an appropriate set of functions for specified tasks and 
user objectives. 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 7 / 47 
 
Accuracy provide the correct or agreed results or effects with the needed 
degree of precision.  
Interoperability interact with one or more specified systems. 
Security protect information and data so that unauthorised persons or 
systems cannot read or modify them and authorised persons or 
systems are not denied access to them. 
Functionality 
compliance 
adhere to standards, conventions or regulations in laws and 
similar prescriptions relating to functionality. 
Reliability maintain a specified level of performance when used under 
specified conditions. 
Maturity avoid failure as a result of faults in the software. 
Fault tolerance maintain a specified level of performance in cases of software 
faults or of infringement of its specified interface. 
Recoverability re-establish a specified level of performance and recover the data 
directly affected in the case of a failure. 
Reliability 
compliance 
adhere to standards, conventions or regulations relating to 
reliability. 
Usability be understood, learned, used and attractive to the user, when 
used under specified conditions. 
Understandability enable the user to understand whether the software is suitable, 
and how it can be used for particular tasks and conditions of use. 
Learnability enable the user to learn its application. 
Operability enable the user to operate and control it. 
Attractiveness be attractive to the user. 
Usability 
compliance 
adhere to standards, conventions, style guides or regulations 
relating to usability. 
Efficiency provide appropriate performance, relative to the amount of 
resources used, under stated conditions.  
Time behaviour provide appropriate response and processing times and 
throughput rates when performing its function, under stated 
conditions. 
Resource utilisation use appropriate amounts and types of resources when the 
software performs its function under stated conditions.  
Efficiency 
compliance 
adhere to standards or conventions relating to efficiency. 
Maintainability be modified. Modifications may include corrections, 
improvements or adaptation of the software to changes in 
environment, and in requirements and functional specifications. 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 8 / 47 
 
Analysability be diagnosed for deficiencies or causes of failures in the 
software, or for the parts to be modified to be identified.  
Changeability enable a specified modification to be implemented. 
Stability avoid unexpected effects from modifications of the software. 
Testability enable modified software to be validated. 
Maintainability 
compliance 
adhere to standards or conventions relating to maintainability. 
Portability be transferred from one environment to another. 
Adaptability be adapted for different specified environments without applying 
actions or means other than those provided for this purpose for 
the software considered. 
Installability be installed in a specified environment. 
Co-existence co-exist with other independent software in a common 
environment sharing common resources. 
Replaceability be used in place of another specified software product for the 
same purpose in the same environment. 
Portability 
compliance 
adhere to standards or conventions relating to portability. 
Table 1: ISO 9126 Quality Attributes. 
Particular attention should be paid to the evaluation of the usability of 
KRSM components. 
 
ISO 9241-11 Guidance on Usability [2] further extends the definition 
of Software Usability. According to ISO 9241-11 (1998), usability is 
the “extent to which a product can be used by specified users to 
achieve specified goals with effectiveness, efficiency and satisfaction in 
a specified context of use.” 
Important aspects of this definition include: 
• “specified users”: It is important to note that when systems are 
being designed for usability, the first step should be identifying 
target user population. Usability is not an absolute term but, rather, 
a relative one. A system can only be usable relative to the user 
population it serves. 
• “specified goals”: The functionality designed into a usable system 
will be relevant to its users. It is often the case that unnecessary or 
inappropriate functionality is incorporated into a system. This 
functionality can “clutter” the interface and make relevant 
functionality more difficult to access. On the other hand, a usable 
 
M5.8 KRSM first cycle prototype evaluation plan 
 
TENCompetence – IST-2005-027087 Page 9 / 47 
 
system presents its users with routes to achieve their goals in a 
clear fashion. 
• “effectiveness”: Effectiveness is the accuracy and completeness 
with which users achieve specified goals. For example, a software 
(SW) is effective if the users can complete tasks making a minimal 
amount of errors. 
• “efficiency”: Efficiency is the resources expended in relation to the 
accuracy and completeness with which users achieve specified 
goals. For example, a SW is efficient if the users are able to achieve 
their goals quickly (saving time) or cheaply (saving money). 
• “satisfaction”: Satisfaction is the freedom from discomfort and 
positive attitudes towards the use of the product. For example, a 
user is satisfied, if (s)he is able to achieve her/his own goals with a 
positive frame of mind. The user may also prefer one particular 
system to other systems. 
• “context of use”: The context of use constitutes the broader 
framework in which a product is operated. It concerns the system’s 
particular users, their tasks and the system’s broader environment 
of use. 
 
2.4 Types of Testing 
 
Testing can be done on a number of different levels: 
• unit/module test; 
• integration test; 
• functional test; 
• system test; 
• acceptance test. 
 
Unit test 
 
Unit testing searches for defects in, and verifies the functioning of, 
software (e.g. modules, programs, objects, classes, etc.) that are 
separately testable. 
Unit tests are typically done by programmers and not by testers, as it 
requires detailed knowledge of the internal program design and code. 
The ideal situation is that another developer then the developer from 
the software runs the unit test. 
 
The purpose of unit testing is to verify that each individual component 
functions according to the technical specifications. Unit testing includes 
several subjects, namely: 


Milestone M5.8 KRSM first cycle prototype evaluation plan

http://dspace.ou.nl/bitstream/1820/975/1/WP5_EvaluationPlan_M5.8_V1.pdf

Milestone M5.8 KRSM first cycle prototype evaluation plan

Abstract

Similar works

Full text

Available Versions

DSpace at Open Universiteit Nederland