Measuring the Quality of Machine Learning and Optimization Frameworks

Software frameworks are daily and extensively used in research, both for fundamental studies and applications. Researchers usually trust in the quality of these frameworks without any evidence that they are correctly build, indeed they could contain some defects that potentially could affect to thousands of already published and future papers. Considering the important role of these frameworks in the current state-of-the-art in research, their quality should be quantified to show the weaknesses and strengths of each software package. In this paper we study the main static quality properties, defined in the product quality model proposed by the ISO 25010 standard, of ten well-known frameworks. We provide a quality rating for each characteristic depending on the severity of the issues detected in the analysis. In addition, we propose an overall quality rating of 12 levels (ranging from A+ to D-) considering the ratings of all characteristics. As a result, we have data evidence to claim that the analysed frameworks are not in a good shape, because the best overall rating is just a C+ for Mahout framework, i.e., all packages need to go for a revision in the analysed features. Focusing on the characteristics individually, maintainability is by far the one which needs the biggest effort to fix the found defects. On the other hand, performance obtains the best average rating, a result which conforms to our expectations because frameworks’ authors used to take care about how fast their software runs.University of Malaga. Campus de Excelencia Internacional Andalucía Tech. We would like to say thank you to all authors of these frameworks that make research easier for all of us. This research has been partially funded by CELTIC C2017/2-2 in collaboration with companies EMERGYA and SECMOTIC with contracts #8.06/5.47.4997 and #8.06/5.47.4996. It has also been funded by the Spanish Ministry of Science and Innovation and /Junta de Andalucı́a/FEDER under contracts TIN2014-57341-R and TIN2017-88213-R, the network of smart cities CI-RTI (TIN2016-81766-REDT

Frameworks for Measuring Innovation: Initial Approaches

In January 2008, the U.S. Department of Commerce released a report, Innovation Measurement: Tracking the State of Innovation in the American Economy (DOC, 2008), which recommended "a stronger framework for identifying and measuring innovation in the national economy." As part of that work, the U.S. Bureau of Economic Analysis (BEA) requested that the Science and Technology Policy Institute (STPI) explore the business perspectives of innovation. The resulting report, Measuring Innovation and Intangibles: A Business Perspective (Stone et al., 2008,) created a compendium of the logic and methods businesses use to measure and monetize innovation. It also identified sources for, and gaps in, innovation data and outlined critical areas for future research. This report extends that work and presents two alternative frameworks for measuring innovation. The first framework focuses on measuring innovation activities at the firm/organization level. The second takes a broader macro-level look at the fundamental investments that allow firms and other organizations to carry out innovation activities

Evaluating three frameworks for the value of information: adaptation to task characteristics and probabilistic structure

We identify, and provide an integration of, three frameworks for measuring the informativeness of cues in a multiple-cue judgment task. Cues can be ranked by information value according to expected information gain (Bayesian framework), cue-outcome correlation (Correlational framework), or ecological validity (Ecological framework). In three experiments, all frameworks significantly predicted information acquisition, with the Correlational (then the Bayesian) framework being most successful. Additionally, participants adapted successfully to task characteristics (cue cost, time pressure, and information limitations) – altering the gross amount of information acquired, but not responding to more subtle features of the cues’ information value that would have been beneficial. Rational analyses of our task environments indicate that participants' behavior can be considered successful from a boundedly rational standpoint

Electrically-Detected ESR in Silicon Nanostructures Inserted in Microcavities

We present the first findings of the new electrically-detected electron spin resonance technique (EDESR), which reveal the point defects in the ultra-narrow silicon quantum wells (Si-QW) confined by the superconductor delta-barriers. This technique allows the ESR identification without application of an external cavity, as well as a high frequency source and recorder, and with measuring the only response of the magnetoresistance, with internal GHz Josephson emission within frameworks of the normal-mode coupling (NMC) caused by the microcavities embedded in the Si-QW plane

Measuring quality in social care services: theory and practice

Measuring and assessing service quality in the social care sector presents distinct challenges. The 'experience' good properties of social care, for instance, and the large influence played by subjective judgements about the quality of personal relationships between carer and user and of process-related service characteristics make it difficult to develop indicators of service quality, including those of service impact on final outcomes. Using some of the key features of the 'Production of Welfare' approach, the paper discusses recent developments in the UK of the theoretical and practical frameworks used for assessing quality in social care and for understanding the final impact of services on the wellbeing of their recipients. Key current and future challenges to the development of such frameworks include difficulties in disentangling the impact of social care services on final outcomes from the often dominating effects of other, non-service related factors, and the generalization of consumer-directed care models and of the 'personalization' of care services. These challenges are discussed in the context of the different possible applications of quality indicators, including their role as supporting the service commissioning process and their use for assessing the performance of service providers

MEASURING SUPPLY-SIDE ECONOMIC IMPACTS ON TOURISM/RECREATION INDUSTRIES

This paper addresses a series of study objectives, starting with (1) the delineation of tourism/recreation activities and providers, (2) the identification of appropriate indicators for measuring economic impact, and (3) the preparation of alternate analytical frameworks for assessing national, regional, and local implications of supply-side economic impacts of tourism/recreation activities. Included, also, among these objectives is (4) the specification of the essential attributes of a public information system for servicing the decision information needs of recreation resource managers.Community/Rural/Urban Development,

Review and analysis of quality healthcare system enhancement in developing countries.

Quality has multidimensional perspectives. It is a continuous and dynamic process. A systematic search of national and international literature was conducted from peer-reviewed databases MEDLINE, CINAHL and PubMed during January to July 2012. There are many frameworks to assess and measure quality. All frameworks give some levels in which quality could be assessed. Therefore, multiple frameworks should be used for measuring quality based on the situation and environment. Measuring quality is very important for primary healthcare programme, especially for developing countries because if we will not do quality assessment, our resources will not be utilised effectively and people may divert to curative services. Thus, in developing countries where healthcare system needs to optimise resources for increased population coverage and scale up the care, quality approaches can suggest workable solutions to using resources appropriately

Tracking Adaptation and Measuring Development in Kenya

Tracking Adaptation and Measuring Development (TAMD) is a twin-track framework that evaluates adaptation success as a combination of how widely and how well countries or institutions manage climate risks (Track 1) and how successful adaptation interventions are in reducing climate vulnerability and in keeping development on course (Track 2). With this twin-track approach, TAMD can be used to assess whether climate change adaptation leads to effective development, and also how development interventions can boost communities' capacity to adapt to climate change. Importantly, TAMD offers a flexible framework that can be used to generate bespoke frameworks for individual countries that can be tailored to specific contexts and used at different scales. This report compiles the results of TAMD feasibility testing phase in Kenya

Measuring Coverage of Prolog Programs Using Mutation Testing

Testing is an important aspect in professional software development, both to avoid and identify bugs as well as to increase maintainability. However, increasing the number of tests beyond a reasonable amount hinders development progress. To decide on the completeness of a test suite, many approaches to assert test coverage have been suggested. Yet, frameworks for logic programs remain scarce. In this paper, we introduce a framework for Prolog programs measuring test coverage using mutations. We elaborate the main ideas of mutation testing and transfer them to logic programs. To do so, we discuss the usefulness of different mutations in the context of Prolog and empirically evaluate them in a new mutation testing framework on different examples.Comment: 16 pages, Accepted for presentation in WFLP 201