An Online Framework for Supporting the Evaluation of Personalised Information Retrieval Systems

Scope - Personalised Information Retrieval (PIR) has been gaining attention because it investigates intelligent ways for enhancing content delivery. Web users can have personalised services and more accurate information. Problem - Several PIR systems have been proposed in the literature; however, they have not been properly tested or evaluated. Proposal – The authors propose a generally applicable web-based interface, which provides PIR developers and evaluators with: i) implicit recommendations on how to evaluate a specific PIR system; ii) a repository containing studies on user-centred and layered evaluation studies; iii) recommendations on how to best combine different evaluation methods, metrics and measurement criteria in order to most effectively evaluate their system; iv) a UCE methodology which details how to apply existing UCE techniques; v) a taxonomy of evaluations of adaptive systems; and vi) interface translation support (49 languages supported)

Evaluating the perception of SaaS adoption criteria with email permission-based marketing

This study aims to identify the main Software as a Service (SaaS) adoption criteria and apply them in permission-based email marketing campaign. Cloud computing and SaaS in precise is becoming a temping business sector for companies. While most of the attention is focus on the technological aspect, the business and marketing sides were less studied. The thesis present an empirical case study to investigate how respondents of small and medium enterprises' (SME), respond to different kind of stimulus from potential service providers, focusing on four distinctive adoption categories. The thesis is empirical and exploratory in nature. In the beginning, cloud computing and SaaS are reviewed to form the background of the study. Afterwards, special attention is dedicated to identifying and grouping the crucial SaaS adoption factors. The Technological-Organization-Environmental (TOE) and Diffusion of Innovations (DOI) frameworks had been used in the process of grouping the adoption factors into four categories. The second part of the literature review is dedicated to permission-based email marketing, examining the positive and negative sides and following how marketing has evolved and let to the adoption of permission-based email marketing. Quantitative data was gathered from a case company which is about to release new SaaS product and which wanted to study the perception of its potential clients towards the four SaaS adoption categories. The findings of the empirical part of this study are separated in four parts. Each part is dedicated to one of the four distinct adoption categories: Usability, Complexity, Security and Price. Usability and Complexity were the best perceived adoption categories while Security and Price didn't attract as much attention as initially anticipated. As a conclusion the study managed to shed some insight about how the SaaS adoption categories are perceived from SME's. The results can't be classified as expected or as absolutely innovative, since two of the categories behaved as it was expected while the other two - Complexity and Security showed some surprising results

Käyttäjäkeskeisen suunnittelun menetelmät uuden ohjelmiston käyttöönotossa organisaatiossa

Organisations that have a software as key component of their business process might run into situation where the used software needs to be upgraded to follow changing business requirements. In such cases the business process of the organisation should be evaluated against the requirements that the new software sets for the process workflow. Also, it is crucial to evaluate how the new software is accepted among the employees of the organisation, because as the end users of the software they are the most affected by the change. User-centered design traditionally has a basic principle that users are involved into the full lifecycle of design and development process. But when a new, existing software is adopted in the organisation, design mainly focuses on the customization of the software. This thesis aims at finding out how user-centered design can be applied when adopting software in an organisation and what changes could be made to the business process of the company so that the software would be utilized in an optimal way. Also, ways of measuring the successfulness of the software adoption in short term is considered. A group walkthrough is used as a method for redefining the business process workflow. Quantitative data about the user selections in the walkthrough is analyzed for understanding the end user interaction with the software and for finding hidden features of the process. Also, qualitative data gathered from the walkthrough is analyzed with thematic analysis method to map out patterns in the workflow that are critical from the point of view of the end user. Redesigned workflow of business process with improvements is introduced as a result of the research. And with thematic analysis, principles of technology acceptance theory are found being in line with the themes raised from the walkthrough data to justify usage of user-centered design. Though user-centered design is found fitting with the technology adoption, iterative process, resourcing users to the test session, and allocating their time for the research purposes can be hard to rationalize as all of it is taken from the contribution of the employees of the company

Standardization of electroencephalography for multi-site, multi-platform and multi-investigator studies: Insights from the canadian biomarker integration network in depression

Subsequent to global initiatives in mapping the human brain and investigations of neurobiological markers for brain disorders, the number of multi-site studies involving the collection and sharing of large volumes of brain data, including electroencephalography (EEG), has been increasing. Among the complexities of conducting multi-site studies and increasing the shelf life of biological data beyond the original study are timely standardization and documentation of relevant study parameters. We presentthe insights gained and guidelines established within the EEG working group of the Canadian Biomarker Integration Network in Depression (CAN-BIND). CAN-BIND is a multi-site, multi-investigator, and multiproject network supported by the Ontario Brain Institute with access to Brain-CODE, an informatics platform that hosts a multitude of biological data across a growing list of brain pathologies. We describe our approaches and insights on documenting and standardizing parameters across the study design, data collection, monitoring, analysis, integration, knowledge-translation, and data archiving phases of CAN-BIND projects. We introduce a custom-built EEG toolbox to track data preprocessing with open-access for the scientific community. We also evaluate the impact of variation in equipment setup on the accuracy of acquired data. Collectively, this work is intended to inspire establishing comprehensive and standardized guidelines for multi-site studies

A Taxonomy of Quality Metrics for Cloud Services

[EN] A large number of metrics with which to assess the quality of cloud services have been proposed over the last years. However, this knowledge is still dispersed, and stakeholders have little or no guidance when choosing metrics that will be suitable to evaluate their cloud services. The objective of this paper is, therefore, to systematically identify, taxonomically classify, and compare existing quality of service (QoS) metrics in the cloud computing domain. We conducted a systematic literature review of 84 studies selected from a set of 4333 studies that were published from 2006 to November 2018. We specifically identified 470 metric operationalizations that were then classified using a taxonomy, which is also introduced in this paper. The data extracted from the metrics were subsequently analyzed using thematic analysis. The findings indicated that most metrics evaluate quality attributes related to performance efficiency (64%) and that there is a need for metrics that evaluate other characteristics, such as security and compatibility. The majority of the metrics are used during the Operation phase of the cloud services and are applied to the running service. Our results also revealed that metrics for cloud services are still in the early stages of maturity only 10% of the metrics had been empirically validated. The proposed taxonomy can be used by practitioners as a guideline when specifying service level objectives or deciding which metric is best suited to the evaluation of their cloud services, and by researchers as a comprehensive quality framework in which to evaluate their approaches.This work was supported by the Spanish Ministry of Science, Innovation and Universities through the Adapt@Cloud Project under Grant TIN2017-84550-R. The work of Ximena Guerron was supported in part by the Universidad Central del Ecuador (UCE), and in part by the Banco Central del Ecuador.Guerron, X.; Abrahao Gonzales, SM.; Insfran, E.; Fernández-Diego, M.; González-Ladrón-De-Guevara, F. (2020). A Taxonomy of Quality Metrics for Cloud Services. IEEE Access. 8:131461-131498. https://doi.org/10.1109/ACCESS.2020.3009079S131461131498

Supporting students' construction of hypermedia

This thesis considers the proposition that hypermedia may be employed effectively in higher education. More specifically, the question of its use by undergraduate music students to assist in writing essays and dissertations is investigated. The work begins with a review of general issues relating to educational hypermedia, such as its history, application, design and architecture. A user-centred approach to hypermedia development is advocated, and after critique and analysis of the literature, a framework for human-computer interaction for educational hypermedia is proposed. A case study is reported which serves to facilitate the undertaking of original research, as well as to evaluate the proposed framework. Other environments are also selected to carry out more generic research. Both reading strategies and writing strategies are investigated, and the results from these studies are used to conduct a repertory grid analysis of students' approaches to and perceptions of essay and dissertation development. The outcome of this experiment concludes with a proposal for a structural model of essay and dissertation development. Analysis of the model suggests the need for further survey analysis of taskartefact usage in specific educational domains, and experimental studies into electronic document manipulation and the reading of music from computer screens are investigated with respect to the case study environment. The implications of the research carried out in this thesis have assisted in and helped to justify the design of the prototype system HECTOR (Hypermedia, from Essay Conception TO Realisation). It aims to support students in their research, planning and writing of essays and dissertations. HECTOR has been evaluated in the field, and the results of this go some way to supporting the hypothesis of the thesis - that hypermedia can be employed effectively in higher education

A Systematic Literature Review of Software Visualization Evaluation

Abstract Context: Software visualizations can help developers to analyze multiple aspects of complex software systems, but their effectiveness is often uncertain due to the lack of evaluation guidelines. Objective: We identify common problems in the evaluation of software visualizations with the goal of formulating guidelines to improve future evaluations. Method: We review the complete literature body of 387 full papers published in the SOFTVIS/VISSOFT conferences, and study 181 of those from which we could extract evaluation strategies, data collection methods, and other aspects of the evaluation. Results: Of the proposed software visualization approaches, 62 lack a strong evaluation. We argue that an effective software visualization should not only boost time and correctness but also recollection, usability, engagement, and other emotions. Conclusion: We call on researchers proposing new software visualizations to provide evidence of their effectiveness by conducting thorough (i) case studies for approaches that must be studied in situ, and when variables can be controlled, (ii) experiments with randomly selected participants of the target audience and real-world open source software systems to promote reproducibility and replicability. We present guidelines to increase the evidence of the effectiveness of software visualization approaches, thus improving their adoption rate