Evaluation campaigns and TRECVid

The TREC Video Retrieval Evaluation (TRECVid) is an international benchmarking activity to encourage research in video information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. TRECVid completed its fifth annual cycle at the end of 2005 and in 2006 TRECVid will involve almost 70 research organizations, universities and other consortia. Throughout its existence, TRECVid has benchmarked both interactive and automatic/manual searching for shots from within a video corpus, automatic detection of a variety of semantic and low-level video features, shot boundary detection and the detection of story boundaries in broadcast TV news. This paper will give an introduction to information retrieval (IR) evaluation from both a user and a system perspective, highlighting that system evaluation is by far the most prevalent type of evaluation carried out. We also include a summary of TRECVid as an example of a system evaluation benchmarking campaign and this allows us to discuss whether such campaigns are a good thing or a bad thing. There are arguments for and against these campaigns and we present some of them in the paper concluding that on balance they have had a very positive impact on research progress

Mission Focused Funding Supporting Nonprofit Sustainability in the Tumult of a Recession

This report is an evaluation of The California Wellness Foundation's (TCWF) grantmaking. Past evaluations of TCWF's core operating report (COS) funding strategy demonstrated that COS has been successful in building organizational sustainability and capacity, as well as helping grantee organizations leverage funds, increase credibility, and build trust with TCWF -- but those studies took place during much better economic times. This report was prepared by Harder+Company Community Research to explore the effects of the COS funding strategies under the far more difficult economic circumstances of 2009 and 2010. Of the 581 grantee organizations with active grants in 2009 and/or 2010, 424 responded to the evaluation survey, reflecting a 73 percent response rate. Through an analysis of TCWF documents and databases, financial data, survey data, and case studies, this report seeks to answer the following evaluation questions: What was the impact of the recession on TCWF grantees?How were the 2009 and 2010 TCWF COS grants used? What was the impact of the 2009 and 2010 COS grants on organizations?By evaluating the COS grants that were active in 2009 and 2010 -- years that include the official end of the recession and its immediate aftermath -- we were able to identify the effects of this type of funding during a period of major financial strain for most nonprofits. This is a unique opportunity to learn about COS that could inform funding practices now and into the future. The findings from this evaluation reveal that in the midst of the recession, TCWF has supported and even strengthened organizations in a variety of important -- and even surprising --ways. This summary highlights key findings, while the full report provides further detail

A data preparation approach for cloud storage based on containerized parallel patterns

In this paper, we present the design, implementation, and evaluation of an efficient data preparation and retrieval approach for cloud storage. The approach includes a deduplication subsystem that indexes the hash of each content to identify duplicated data. As a consequence, avoiding duplicated content reduces reprocessing time during uploads and other costs related to outsource data management tasks. Our proposed data preparation scheme enables organizations to add properties such as security, reliability, and cost-efficiency to their contents before sending them to the cloud. It also creates recovery schemes for organizations to share preprocessed contents with partners and end-users. The approach also includes an engine that encapsulates preprocessing applications into virtual containers (VCs) to create parallel patterns that improve the efficiency of data preparation retrieval process. In a study case, real repositories of satellite images, and organizational files were prepared to be migrated to the cloud by using processes such as compression, encryption, encoding for fault tolerance, and access control. The experimental evaluation revealed the feasibility of using a data preparation approach for organizations to mitigate risks that still could arise in the cloud. It also revealed the efficiency of the deduplication process to reduce data preparation tasks and the efficacy of parallel patterns to improve the end-user service experience.This research was supported by "Fondo Sectorial de Investigación para la Educación";, SEP-CONACyT Mexico, through projects 281565 and 285276

NLP Methods in Host-based Intrusion Detection Systems: A Systematic Review and Future Directions

Host based Intrusion Detection System (HIDS) is an effective last line of defense for defending against cyber security attacks after perimeter defenses (e.g., Network based Intrusion Detection System and Firewall) have failed or been bypassed. HIDS is widely adopted in the industry as HIDS is ranked among the top two most used security tools by Security Operation Centers (SOC) of organizations. Although effective and efficient HIDS is highly desirable for industrial organizations, the evolution of increasingly complex attack patterns causes several challenges resulting in performance degradation of HIDS (e.g., high false alert rate creating alert fatigue for SOC staff). Since Natural Language Processing (NLP) methods are better suited for identifying complex attack patterns, an increasing number of HIDS are leveraging the advances in NLP that have shown effective and efficient performance in precisely detecting low footprint, zero day attacks and predicting the next steps of attackers. This active research trend of using NLP in HIDS demands a synthesized and comprehensive body of knowledge of NLP based HIDS. Thus, we conducted a systematic review of the literature on the end to end pipeline of the use of NLP in HIDS development. For the end to end NLP based HIDS development pipeline, we identify, taxonomically categorize and systematically compare the state of the art of NLP methods usage in HIDS, attacks detected by these NLP methods, datasets and evaluation metrics which are used to evaluate the NLP based HIDS. We highlight the relevant prevalent practices, considerations, advantages and limitations to support the HIDS developers. We also outline the future research directions for the NLP based HIDS development

Towards An Optimal Level Of Participation Of The Intermediary In The User-system Interface Of Bibliographic Online Search Services

The present research project focuses on the user/system interface of on-line bibliographic search services and more specifically on the role of the human intermediary in the context of on-line searching. A quasi-experimental study was designed with 34 end-users and 22 intermediaries who performed 102 online searches in 6 different organizations (3 academic and 3 special libraries). Each end-user was given a one-day training session and then prepared three different search questions. An on-line search was performed for each question: one direct search (the end-user alone), one delegated search (the intermediary alone) and one combined search (the end-user and intermediary together).;Numerous characteristics of end-users, intermediaries and search questions were assessed. Nine outcome measures were recorded from evaluation questionnaires, printouts and output citations assessed for relevance. Results can be summarized as follows: The direct mode was found superior for total search time, overall user satisfaction and three specific satisfaction measures (number of citations, search mode, search utility), search efficiency, cost effectiveness, cost efficiency and search effort. The delegated mode showed best results on total search cost, connect time, connect cost, search effectiveness and searching efficiency. Finally, the combined mode was best for user satisfaction with the relevance of citations retrieved, and search utility.;Discriminant functions were computed to predict the search mode which would optimize each of the nine performance measures. The best predictor variables were: end-user\u27s age, one specific search purpose (e.g. preparing a class project or term paper), the previous use of secondary journals (indexes or abstracts) by end-users, the previous use of the search service center by end-users, and end-users\u27 experience (level) with online searching

An MADM risk-based evaluation-selection model of free-libre open source software tools

Free-Libre Open Source Software (FLOSS) tools are free-cost license highly attractive to be implemented by organizations. However, not of all the FLOSS tools are mature, and failed implementations can occur. Thus, FLOSS evaluation-selection frameworks and FLOSS success-failure implementation factors studies have been conducted. In this research, we advance on such studies through an integrated FLOSS evaluation-selection model with a risk-based decision making approach. Our model was built upon the other two literatures, and it was structured as a Multi-Attribute Decision Making (MADM) model which contains 12 variables grouped in four risk categories: financial, organizational, end-user and technical ones. We illustrated its utilization in the domain of Information Technology Service Management (ITSM) FLOSS tools. Hence, our model contributes to the FLOSS literature with the inclusion of the risk management approach and to the FLOSS evaluation-selection praxis with the provision of an innovative and essential risk-based model

Refining policy with the poor: local consultations on the draft comprehensive poverty reduction and growth strategy in Vietnam

In March 2001, the Government of Vietnam produced an Interim Poverty Reduction Strategy Paper (I-PRSP), and announced its intention to develop a Comprehensive Poverty Reduction Strategy Paper (CPRGS) by the end of April 2002. In the I-PRSP, the Government outlined its commitment to involve a broad range of stakeholders - including poor communities, local government authorities, and the enterprise sector - in drafting the CPRGS. The Ministry of Planning and Investment, who was assigned by the Government of Vietnam to lead the CPRGS drafting process, asked the World Bank, and a group of international nongovernmental organizations to support them in carrying out the local consultations. The consultations took place in six rural, and urban locations across Vietnam, selected to represent a range of poverty situations. About 1,800 people participated in the research. This report, which is the first of three volumes documenting the local consultations, provides an account and reflection on the approach, and methodology used in the consultations. It is intended this may give useful practical experience for future monitoring of the CPRGS, as well as for people who are planning to carry out similar exercises in other countries. The report outlines the process that was followed from the point of developing a research outline from the I-PRSP, through the fieldwork exercises, data compilation, and analysis, leading on to identification of the main policy messages made by the participants. It also describes how the findings were used to influence the final version of the CPRGS.Poverty Monitoring&Analysis,Public Health Promotion,ICT Policy and Strategies,Health Monitoring&Evaluation,Poverty Reduction Strategies,Poverty Assessment,Health Monitoring&Evaluation,Poverty Monitoring&Analysis,Achieving Shared Growth,Urban Partnerships&Poverty

Towards a consensus around standards for smartphone apps and digital mental health

Mental disorders impact one in four people worldwide, yet access to care is challenging for those who suffer from them1. Mental health apps offer the potential to overcome access barriers for the nearly three billion people projected to own a smartphone by 2020. Although there are over 10,000 mental health apps commercially available, there are few resources available to help end users (patients, clinicians and health care organizations) to evaluate the quality and suitability of these products. Thus, there is an urgent need for an agreement about appropriate standards, principles and practices in research and evaluation of these tools.We represent leaders in mHealth research, industry and health care systems from around the globe, and we seek here to promote consensus on implementing these standards and principles for the evaluation of mental health apps. At a minimum, standards should include consideration of: a) data safety and privacy, b) effectiveness, c) user experience/adherence, d) data integration. Our consensus on the challenges and recommendations in each of these areas is presented below