Relevance thresholds in system evaluations

We introduce and explore the concept of an individual's relevance threshold as a way of reconciling differences in outcomes between batch and user experiments

Improving Ontology Recommendation and Reuse in WebCORE by Collaborative Assessments

In this work, we present an extension of CORE [8], a tool for Collaborative Ontology Reuse and Evaluation. The system receives an informal description of a specific semantic domain and determines which ontologies from a repository are the most appropriate to describe the given domain. For this task, the environment is divided into three modules. The first component receives the problem description as a set of terms, and allows the user to refine and enlarge it using WordNet. The second module applies multiple automatic criteria to evaluate the ontologies of the repository, and determines which ones fit best the problem description. A ranked list of ontologies is returned for each criterion, and the lists are combined by means of rank fusion techniques. Finally, the third component uses manual user evaluations in order to incorporate a human, collaborative assessment of the ontologies. The new version of the system incorporates several novelties, such as its implementation as a web application; the incorporation of a NLP module to manage the problem definitions; modifications on the automatic ontology retrieval strategies; and a collaborative framework to find potential relevant terms according to previous user queries. Finally, we present some early experiments on ontology retrieval and evaluation, showing the benefits of our system

Endoscopic scores for inflammatory bowel disease in the era of 'mucosal healing'. old problem, new perspectives

The importance of the endoscopic evaluation in inflammatory bowel disease (IBD) management has been recognized for many years. However, the modalities for reporting endoscopic activity represent an ongoing challenge. To address this, several endoscopic scores have been proposed. Very few have been properly validated, and the use of such tools remains sub-optimal and is mainly restricted to clinical trials. In recent years, a growing emphasis of the concept of 'mucosal healing' as a prognostic marker and therapeutic goal has increased the need for a more accurate definition of endoscopic activity in both ulcerative colitis (UC) and Crohn's Disease (CD). In the present review, the evolution of the challenges related to endoscopic scores in IBD has been analyzed, with particular attention paid to the renewed relevance of endoscopic activity in recent years. Currently, despite the growing relevance of endoscopic activity, evaluating this activity in IBD is still a challenge. The implementation of efficacious endoscopic scores and a better definition of the absence of activity (mucosal healing) are needed.The importance of the endoscopic evaluation in inflammatory bowel disease (IBD) management has been recognized for many years. However, the modalities for reporting endoscopic activity represent an ongoing challenge. To address this, several endoscopic scores have been proposed. Very few have been properly validated, and the use of such tools remains sub-optimal and is mainly restricted to clinical trials. In recent years, a growing emphasis of the concept of 'mucosal healing' as a prognostic marker and therapeutic goal has increased the need for a more accurate definition of endoscopic activity in both ulcerative colitis (UC) and Crohn's Disease (CD). In the present review, the evolution of the challenges related to endoscopic scores in IBD has been analyzed, with particular attention paid to the renewed relevance of endoscopic activity in recent years. Currently, despite the growing relevance of endoscopic activity, evaluating this activity in IBD is still a challenge. The implementation of efficacious endoscopic scores and a better definition of the absence of activity (mucosal healing) are needed

Do not be afraid of local minima: affine shaker and particle swarm

Stochastic local search techniques are powerful and flexible methods to optimize difficult functions. While each method is characterized by search trajectories produced through a randomized selection of the next step, a notable difference is caused by the interaction of different searchers, as exemplified by the Particle Swarm methods. In this paper we evaluate two extreme approaches, Particle Swarm Optimization, with interaction between the individual "cognitive" component and the "social" knowledge, and Repeated Affine Shaker, without any interaction between searchers but with an aggressive capability of scouting out local minima. The results, unexpected to the authors, show that Affine Shaker provides remarkably efficient and effective results when compared with PSO, while the advantage of Particle Swarm is visible only for functions with a very regular structure of the local minima leading to the global optimum and only for specific experimental conditions

From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking

Objectives Resource allocation is a challenging issue faced by health policy decisionmakers requiring careful consideration of many factors. Objectives of this study were to identify decision criteria and their frequency reported in the literature on healthcare decisionmaking. Method An extensive literature search was performed in Medline and EMBASE to identify articles reporting healthcare decision criteria. Studies conducted with decisionmakers (e.g., focus groups, surveys, interviews), conceptual and review articles and articles describing multicriteria tools were included. Criteria were extracted, organized using a classification system derived from the EVIDEM framework and applying multicriteria decision analysis (MCDA) principles, and the frequency of their occurrence was measured. Results Out of 3146 records identified, 2790 were excluded. Out of 356 articles assessed for eligibility, 40 studies included. Criteria were identified from studies performed in several regions of the world involving decisionmakers at micro, meso and macro levels of decision and from studies reporting on multicriteria tools. Large variations in terminology used to define criteria were observed and 360 different terms were identified. These were assigned to 58 criteria which were classified in 9 different categories including: health outcomes; types of benefit; disease impact; therapeutic context; economic impact; quality of evidence; implementation complexity; priority, fairness and ethics; and overall context. The most frequently mentioned criteria were: equity/fairness (32 times), efficacy/effectiveness (29), stakeholder interests and pressures (28), cost-effectiveness (23), strength of evidence (20), safety (19), mission and mandate of health system (19), organizational requirements and capacity (17), patient-reported outcomes (17) and need (16). Conclusion This study highlights the importance of considering both normative and feasibility criteria for fair allocation of resources and optimized decisionmaking for coverage and use of healthcare interventions. This analysis provides a foundation to develop a questionnaire for an international survey of decisionmakers on criteria and their relative importance. The ultimate objective is to develop sound multicriteria approaches to enlighten healthcare decisionmaking and priority-settin

LexRank: Graph-based Lexical Centrality as Salience in Text Summarization

We introduce a stochastic graph-based method for computing relative importance of textual units for Natural Language Processing. We test the technique on the problem of Text Summarization (TS). Extractive TS relies on the concept of sentence salience to identify the most important sentences in a document or set of documents. Salience is typically defined in terms of the presence of particular important words or in terms of similarity to a centroid pseudo-sentence. We consider a new approach, LexRank, for computing sentence importance based on the concept of eigenvector centrality in a graph representation of sentences. In this model, a connectivity matrix based on intra-sentence cosine similarity is used as the adjacency matrix of the graph representation of sentences. Our system, based on LexRank ranked in first place in more than one task in the recent DUC 2004 evaluation. In this paper we present a detailed analysis of our approach and apply it to a larger data set including data from earlier DUC evaluations. We discuss several methods to compute centrality using the similarity graph. The results show that degree-based methods (including LexRank) outperform both centroid-based methods and other systems participating in DUC in most of the cases. Furthermore, the LexRank with threshold method outperforms the other degree-based techniques including continuous LexRank. We also show that our approach is quite insensitive to the noise in the data that may result from an imperfect topical clustering of documents