Soft peer review: social software and distributed scientific evaluation

The debate on the prospects of peer-review in the Internet age and the increasing criticism leveled against the dominant role of impact factor indicators are calling for new measurable criteria to assess scientific quality. Usage-based metrics offer a new avenue to scientific quality assessment but face the same risks as first generation search engines that used unreliable metrics (such as raw traffic data) to estimate content quality. In this article I analyze the contribution that social bookmarking systems can provide to the problem of usage-based metrics for scientific evaluation. I suggest that collaboratively aggregated metadata may help fill the gap between traditional citation-based criteria and raw usage factors. I submit that bottom-up, distributed evaluation models such as those afforded by social bookmarking will challenge more traditional quality assessment models in terms of coverage, efficiency and scalability. Services aggregating user-related quality indicators for online scientific content will come to occupy a key function in the scholarly communication system

Peer review innovations in Humanities: how can scholars in A&H profit of the "wisdom of the crowds"?

Though supported by a large number of scholars in Scientific, Technical, and Medical (STM) disciplines traditional peer review does not live up to the needs of an efficient scholarly communication system and of quality research control. Therefore journals in STM are experimenting different forms of refereeing in combination with more traditional peer review system. Such is the case of PLoSONE, Biology Direct, Atmospheric Chemistry and Physics, Electronic Transactions on Artificial Intelligence, and JIME. However in STM disciplines public peer review is not regarded an alternative to more traditional quality certification forms. It may be the case in the Arts & Humanities. In A&H publishing system peer review is by far a less common practice. Therefore the adoption of a social peer review process could be very useful to foster research in humanities. Scholars in A&H can profit of the interactive evaluation forms of the public peer-review to strengthen the scholarly debate, to foster active international and interdisciplinary discussions, to focus social attention on topics in Humanities, to broaden the borders of the cultural and intellectual discourse among non-scholars (public debate). This paper will provide some examples of how social peer review has been adopted by innovative communities of scholars in humanities to publish new experimental digital book models. In the digital environment the concepts of “document”, of “completeness of a document” and of “evaluation” is fast changing. In a close future in scholarly publishing it might become possible to overcome the rigid distinction between ex-ante and ex-post evaluation as the evaluation process might become an enduring part of the text itsel

Summary of the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1)

Challenges related to development, deployment, and maintenance of reusable software for science are becoming a growing concern. Many scientists’ research increasingly depends on the quality and availability of software upon which their works are built. To highlight some of these issues and share experiences, the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1) was held in November 2013 in conjunction with the SC13 Conference. The workshop featured keynote presentations and a large number (54) of solicited extended abstracts that were grouped into three themes and presented via panels. A set of collaborative notes of the presentations and discussion was taken during the workshop. Unique perspectives were captured about issues such as comprehensive documentation, development and deployment practices, software licenses and career paths for developers. Attribution systems that account for evidence of software contribution and impact were also discussed. These include mechanisms such as Digital Object Identifiers, publication of “software papers”, and the use of online systems, for example source code repositories like GitHub. This paper summarizes the issues and shared experiences that were discussed, including cross-cutting issues and use cases. It joins a nascent literature seeking to understand what drives software work in science, and how it is impacted by the reward systems of science. These incentives can determine the extent to which developers are motivated to build software for the long-term, for the use of others, and whether to work collaboratively or separately. It also explores community building, leadership, and dynamics in relation to successful scientific software

Skills and Knowledge for Data-Intensive Environmental Research.

The scale and magnitude of complex and pressing environmental issues lend urgency to the need for integrative and reproducible analysis and synthesis, facilitated by data-intensive research approaches. However, the recent pace of technological change has been such that appropriate skills to accomplish data-intensive research are lacking among environmental scientists, who more than ever need greater access to training and mentorship in computational skills. Here, we provide a roadmap for raising data competencies of current and next-generation environmental researchers by describing the concepts and skills needed for effectively engaging with the heterogeneous, distributed, and rapidly growing volumes of available data. We articulate five key skills: (1) data management and processing, (2) analysis, (3) software skills for science, (4) visualization, and (5) communication methods for collaboration and dissemination. We provide an overview of the current suite of training initiatives available to environmental scientists and models for closing the skill-transfer gap

Report on the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2)

This technical report records and discusses the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2). The report includes a description of the alternative, experimental submission and review process, two workshop keynote presentations, a series of lightning talks, a discussion on sustainability, and five discussions from the topic areas of exploring sustainability; software development experiences; credit & incentives; reproducibility & reuse & sharing; and code testing & code review. For each topic, the report includes a list of tangible actions that were proposed and that would lead to potential change. The workshop recognized that reliance on scientific software is pervasive in all areas of world-leading research today. The workshop participants then proceeded to explore different perspectives on the concept of sustainability. Key enablers and barriers of sustainable scientific software were identified from their experiences. In addition, recommendations with new requirements such as software credit files and software prize frameworks were outlined for improving practices in sustainable software engineering. There was also broad consensus that formal training in software development or engineering was rare among the practitioners. Significant strides need to be made in building a sense of community via training in software and technical practices, on increasing their size and scope, and on better integrating them directly into graduate education programs. Finally, journals can define and publish policies to improve reproducibility, whereas reviewers can insist that authors provide sufficient information and access to data and software to allow them reproduce the results in the paper. Hence a list of criteria is compiled for journals to provide to reviewers so as to make it easier to review software submitted for publication as a “Software Paper.

IDR : a participatory methodology for interdisciplinary design in technology enhanced learning

One of the important themes that emerged from the CAL’07 conference was the failure of technology to bring about the expected disruptive effect to learning and teaching. We identify one of the causes as an inherent weakness in prevalent development methodologies. While the problem of designing technology for learning is irreducibly multi-dimensional, design processes often lack true interdisciplinarity. To address this problem we present IDR, a participatory methodology for interdisciplinary techno-pedagogical design, drawing on the design patterns tradition (Alexander, Silverstein & Ishikawa, 1977) and the design research paradigm (DiSessa & Cobb, 2004). We discuss the iterative development and use of our methodology by a pan-European project team of educational researchers, software developers and teachers. We reflect on our experiences of the participatory nature of pattern design and discuss how, as a distributed team, we developed a set of over 120 design patterns, created using our freely available open source web toolkit. Furthermore, we detail how our methodology is applicable to the wider community through a workshop model, which has been run and iteratively refined at five major international conferences, involving over 200 participants

Sustainability in Software Engineering

The intersection between software engineering re- search and the problems related to sustainability and green IT has been the subject of increasing attention. In spite of that, we observe that sustainability is still not clearly defined, or understood, in the field of software engineering. This lack of clarity leads to confusion about e.g. what is relevant to measure or the research implications over time or space. This paper provides an overview of how the research so far has defined sustainability, and how this definition has been used to guide which research areas. To this end, we carried out a systematic mapping study for selecting, classifying and analyzing relevant publications. In this study, we investigate which knowledge areas and which time scope of sustainability effects are mostly targeted in scientific research. Our analysis shows research trends and discusses gaps to be filled

Astrophysicists and physicists as creators of ArXiv-based commenting resources for their research communities. An initial survey

This paper conveys the outcomes of what results to be the first, though initial, overview of commenting platforms and related 2.0 resources born within and for the astrophysical community (from 2004 to 2016). Experiences were added, mainly in the physics domain, for a total of 22 major items, including four epijournals, and four supplementary resources, thus casting some light onto an unexpected richness and consonance of endeavours. These experiences rest almost entirely on the contents of the database ArXiv, which adds to its merits that of potentially setting the grounds for web 2.0 resources, and research behaviours, to be explored. Most of the experiences retrieved are UK and US based, but the resulting picture is international, as various European countries, China and Australia have been actively involved. Final remarks about creation patterns and outcome of these resources are outlined. The results integrate the previous studies according to which the web 2.0 is presently of limited use for communication in astrophysics and vouch for a role of researchers in the shaping of their own professional communication tools that is greater than expected. Collaterally, some aspects of ArXiv s recent pathway towards partial inclusion of web 2.0 features are touched upon. Further investigation is hoped for.Comment: Journal article 16 page