Evaluation campaigns and TRECVid

The TREC Video Retrieval Evaluation (TRECVid) is an international benchmarking activity to encourage research in video information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. TRECVid completed its fifth annual cycle at the end of 2005 and in 2006 TRECVid will involve almost 70 research organizations, universities and other consortia. Throughout its existence, TRECVid has benchmarked both interactive and automatic/manual searching for shots from within a video corpus, automatic detection of a variety of semantic and low-level video features, shot boundary detection and the detection of story boundaries in broadcast TV news. This paper will give an introduction to information retrieval (IR) evaluation from both a user and a system perspective, highlighting that system evaluation is by far the most prevalent type of evaluation carried out. We also include a summary of TRECVid as an example of a system evaluation benchmarking campaign and this allows us to discuss whether such campaigns are a good thing or a bad thing. There are arguments for and against these campaigns and we present some of them in the paper concluding that on balance they have had a very positive impact on research progress

Saudi Accented Arabic Voice Bank

AbstractThe aim of this paper is to present an Arabic speech database that represents Arabic native speakers from all the cities of Saudi Arabia. The database is called the Saudi Accented Arabic Voice Bank (SAAVB). Preparing the prompt sheets, selecting the right speakers and transcribing their speech are some of the challenges that faced the project team. The procedures that meet these challenges are highlighted. SAAVB consists of 1033 speakers speak in Modern Standard Arabic with a Saudi accent. The SAAVB content is analyzed and the results are illustrated. The content was verified internally and externally by IBM Cairo and can be used to train speech engines such as automatic speech recognition and speaker verification systems

Competitive Evaluation of Commercially Available Speech Recognizers in Multiple Languages

Recent improvements in speech recognition technology have resulted in products that can now demonstrate commercial value in a variety of applications. Many vendors are marketing products which combine ASR applications including continuous dictation, command-and-control interfaces, and transcription of recorded speech at an accuracy of 98%. In this study, we measured the accuracy of certain commercially available desktop speech recognition engines in multiple languages. Using word error rate as a benchmark, this work compares recognition accuracy across eight languages and the products of three manufacturers. Results show that two systems performed almost the same while a third system recognized at lower accuracy, although none of the systems reached the claimed accuracy. Read speech was recognized better than spontaneous speech. The systems for US-English, Japanese and Spanish showed higher accuracy than the systems for UK-English, German, French and Chinese

Big Data Privacy Context: Literature Effects On Secure Informational Assets

This article's objective is the identification of research opportunities in the current big data privacy domain, evaluating literature effects on secure informational assets. Until now, no study has analyzed such relation. Its results can foster science, technologies and businesses. To achieve these objectives, a big data privacy Systematic Literature Review (SLR) is performed on the main scientific peer reviewed journals in Scopus database. Bibliometrics and text mining analysis complement the SLR. This study provides support to big data privacy researchers on: most and least researched themes, research novelty, most cited works and authors, themes evolution through time and many others. In addition, TOPSIS and VIKOR ranks were developed to evaluate literature effects versus informational assets indicators. Secure Internet Servers (SIS) was chosen as decision criteria. Results show that big data privacy literature is strongly focused on computational aspects. However, individuals, societies, organizations and governments face a technological change that has just started to be investigated, with growing concerns on law and regulation aspects. TOPSIS and VIKOR Ranks differed in several positions and the only consistent country between literature and SIS adoption is the United States. Countries in the lowest ranking positions represent future research opportunities.Comment: 21 pages, 9 figure

Popular and/or Prestigious? Measures of Scholarly Esteem

Citation analysis does not generally take the quality of citations into account: all citations are weighted equally irrespective of source. However, a scholar may be highly cited but not highly regarded: popularity and prestige are not identical measures of esteem. In this study we define popularity as the number of times an author is cited and prestige as the number of times an author is cited by highly cited papers. Information Retrieval (IR) is the test field. We compare the 40 leading researchers in terms of their popularity and prestige over time. Some authors are ranked high on prestige but not on popularity, while others are ranked high on popularity but not on prestige. We also relate measures of popularity and prestige to date of Ph.D. award, number of key publications, organizational affiliation, receipt of prizes/honors, and gender.Comment: 26 pages, 5 figure

Rethinking the Dutch Innovation Agenda: Management and Organization Matter Most

In this essay, we challenge the present dominant emphasis in the Dutch Innovation Debate on the creation of technological innovations, the focus on a few core technologies, and the allocation of more financial resources. We argue that managerial capabilities and organizing principles for innovation should have a higher priority on the Dutch Innovation Agenda. Managerial capabilities for innovation deal with cognitive elements such as the capacity to absorb knowledge, create entrepreneurial mindsets, and facilitate managerial experimentation and higher-order learning abilities. These capacities can only be developed by distinctive managerial roles that enhance hierarchy, teaming and shared norms. Utilizing these unique managerial capabilities requires novel organizing principles, such as managing internal rates of change, nurturing self-organization and balancing high levels of exploration and exploitation. These managerial capabilities and organizing principles of innovation create new sources of productivity growth and competitive advantage.The dramatic fall back of the Netherlands in the league of innovative and high productivity countries of the World Economic Forum-Report can be mainly attributed to the present lack in the Netherlands of these key managerial and organizational enablers of innovation and productivity growth. We provide various levers for building unique managerial capabilities and novel organizing principles of innovation. Moreover, we describe the necessary roles that different actors have to play in this innovation arena. In particular, we focus on the often neglected but important role of strategic regulations that speed up innovation and productivity growth. They are the least expensive way to boost innovation in organizations in both the Dutch private and public sector. Finally, we discuss the implications for the Dutch Innovation Agenda. It should start with setting a challenging ambition, namely the return of The Netherlands within the WEF- league of the top-ten most innovative and productive countries of the world. Considering the under-utilization of available knowledge stemming from technological innovations, managerial and organizational determinants of innovation should receive first priority. These determinants have a high strategic relevance and should receive more public recognition. We suggest to organize an annual innovation ranking of the most outstanding Dutch firms, to develop an innovation audit that measures firms’ non-technological innovation capacity, and to create an overall innovation policy for fast diffusion of new managerial capabilities and adequate organizing principles throughout the Dutch private and public sector.In conclusion, we add five new items to the Dutch Innovation Agenda:1. Prioritize administrative innovationsInvestments in management and organization determinants of absorption of knowledge and its successful application (administrative innovation) should have a higher priority than investments in technological innovations.2. Build new managerial capabilities and develop novel organizing principlesFor these administrative innovations to succeed, firms have to build managerial capabilities (broad knowledge-base, absorptive capacity, managerial experimentation, higher-order learning) and various management roles (hierarchy, teaming, shared norms) to increase the assimilation of external knowledge and the utilization for innovation. Moreover, they have to develop novel organizing principles that increase internal rates of change, nurture self-organization and synchronize high levels of exploration and exploitation.3. Set levers of innovation by creating selection environments that favor innovation and by redefining the roles of key actors Management has to create a proper organizational context to foster entrepreneurship and innovation (internal selection environment). Governmental agencies have to focus on innovation and productivity enabling strategic regulations (external selection environment). Moreover, research institutes, business schools, and consulting firms should not only focus on technological knowledge, but also on managerial and organizational knowledge for innovation. In the end, private small and large firms and public institutions have to recognize that they all must contribute to the national goal of increasing innovation and productivity growth.4. Create a new challenging national ambition: return of the Netherlands within the top-10The Netherlands has to return to the top-ten most innovative and productive countries in the world as reflected in international rankings such as the World Economic Forum’s Global Competitiveness Index.5. Proliferate an awareness and passion for innovation:Create public awareness and recognition of the societal relevance of outstanding managerial capabilities and organizing principles to innovation and productivity growth:o Initiate a Dutch innovation ranking in terms of management and organization;o Develop proper assessment tools for innovations in management and organization;o Enhance reporting on the progress on managerial and organizational innovation as part of modern corporate governance and as part of outstanding annual reports.These issues may contribute to rethinking the fundamental sources of innovation, productivity growth and sustainable competitive advantage of the Dutch economy.dynamic capabilities;knowledge transfer;exploitation;exploration;mANAGEMENT;mindsets;organizing pinciples;srategic rgulation;strategy innovation