Benchmarking: A methodology for ensuring the relative quality of recommendation systems in software engineering

This chapter describes the concepts involved in the process of benchmarking of recommendation systems. Benchmarking of recommendation systems is used to ensure the quality of a research system or production system in comparison to other systems, whether algorithmically, infrastructurally, or according to any sought-after quality. Specifically, the chapter presents evaluation of recommendation systems according to recommendation accuracy, technical constraints, and business values in the context of a multi-dimensional benchmarking and evaluation model encompassing any number of qualities into a final comparable metric. The focus is put on quality measures related to recommendation accuracy, technical factors, and business values. The chapter first introduces concepts related to evaluation and benchmarking of recommendation systems, continues with an overview of the current state of the art, then presents the multi-dimensional approach in detail. The chapter concludes with a brief discussion of the introduced concepts and a summary

MLPerf Inference Benchmark

Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in an architecture-neutral, representative, and reproducible manner challenging. There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call. In this paper, we present our benchmarking method for evaluating ML inference systems. Driven by more than 30 organizations as well as more than 200 ML engineers and practitioners, MLPerf prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures. The first call for submissions garnered more than 600 reproducible inference-performance measurements from 14 organizations, representing over 30 systems that showcase a wide range of capabilities. The submissions attest to the benchmark's flexibility and adaptability.Comment: ISCA 202

Towards the improvement of the Semantic Web technology

The Semantic Web technology needs to be thoroughly evalu- ated for providing objective results and to attain a massive improvement in their quality in order to be consolidated in the industrial and in the academic world. This paper presents software benchmarking as a process to carry out over the SemanticWeb technology in order to improve it and to search for best practices. It also describes a software benchmarking methodology and provides recommendations for performing evaluations in benchmarking activities

Benchmarking in the Semantic Web

The Semantic Web technology needs to be thoroughly evaluated for providing objective results and obtaining massive improvement in its quality; thus, the transfer of this technology from research to industry will speed up. This chapter presents software benchmarking, a process that aims to improve the Semantic Web technology and to find the best practices. The chapter also describes a specific software benchmarking methodology and shows how this methodology has been used to benchmark the interoperability of ontology development tools, employing RDF(S) as the interchange language

Benchmarking to improve efficiency Status Report

HESA was commissioned by HEFCE to provide an assimilation of current activity within the UK HE sector in relation to benchmarking, under the title of Benchmarking to improve efficiency – Status Report. This was envisaged as a first phase project to draw together information on available and potential data sources and services for benchmarking, produce an inventory of benchmarking activities across the sector and generate some more indepth case studies of selected benchmarking initiatives. It was envisaged that this would point the way to a second phase project which would aim to improve and increase benchmarking capacity and capability in the sector to support increasing efficiencies. The project team conducted a rapid appraisal of benchmarking data, activities and research, against a challenging timescale. The HESA HEI User Group, which includes representation from a broad range of sector associations, acted in a steering capacity. Information was gathered through contact with relevant HE representative bodies, funding bodies and data providers, focused around the HESA HEI User Group but supplemented where appropriate by a range of other contacts. Semi‐structured interviews were held by telephone or in person with members of staff at HEIs and key organisations who are involved in benchmarking activities and initiatives. Key information was gathered by means of a questionnaire to the planning community. Reference was also made to academic and other studies on benchmarking

CHORUS Deliverable 2.2: Second report - identification of multi-disciplinary key issues for gap analysis toward EU multimedia search engines roadmap

After addressing the state-of-the-art during the first year of Chorus and establishing the existing landscape in multimedia search engines, we have identified and analyzed gaps within European research effort during our second year. In this period we focused on three directions, notably technological issues, user-centred issues and use-cases and socio- economic and legal aspects. These were assessed by two central studies: firstly, a concerted vision of functional breakdown of generic multimedia search engine, and secondly, a representative use-cases descriptions with the related discussion on requirement for technological challenges. Both studies have been carried out in cooperation and consultation with the community at large through EC concertation meetings (multimedia search engines cluster), several meetings with our Think-Tank, presentations in international conferences, and surveys addressed to EU projects coordinators as well as National initiatives coordinators. Based on the obtained feedback we identified two types of gaps, namely core technological gaps that involve research challenges, and “enablers”, which are not necessarily technical research challenges, but have impact on innovation progress. New socio-economic trends are presented as well as emerging legal challenges

Energy Information Systems: From the Basement to the Boardroom

A significant buildings energy reduction opportunity exists in the office sector, given that this market segment typically is an early adopter of new technology. There is a rising trend towards smart and connected offices through the internet of things (IoT) that provides new opportunities for operational efficiency and environmental sustainability practices. Leading commercial real estate companies have begun to shift from individual building automation systems (BAS) to partially integrated and automated systems such as energy information systems (EIS). In both the United States and India, organizations are seeking operational excellence, enhanced tenant relationships, and topline growth. Hence it is imperative to engage the executives with decision-making power, by tapping into their interest in sustainability, corporate social responsibility, and innovation. This expansion of interest can enable data-driven decisions, strong energy investments, and deeper energy benefits, and would drive innovation in this field. However, none of this would be possible without robust, consistent building energy information to provide visibility across all the levels of decision making, i.e. from the basement where the facilities staff take operational action to the boardroom where the executives make investment decisions. Price, security, and ease of use remain barriers to the adoption and pervasive use of promising EIS technologies in commercial office buildings. We believe that these barriers can be addressed through the development of ready, simplified, consistent, commercially available, low-cost EIS-in-a-box packages, that have a pre-defined set of hardware components and software features and functionality that are pertinent to a particular building sector. These simplified, sector-specific EIS packages can help to obviate the need for customization, and enhance ease of use, thereby enabling scale-up, in order to facilitate building energy savings. The EIS-in-a-box are adaptable in both U.S. and Indian office buildings, and potentially beyond these two countries

CASE STUDY ON SAFETY MANAGEMENT IN CONSTRUCTION SITE: Proposed 40 units 3-Storey Shop Office & 4 Units Of 2-Storey Shop Office

There have been significant reductions in the number and the rate of injury over the last 20 years or more. Nevertheless, construction remains as the one of the high risk industry. The purpose of this study is to examine safety management in the Malaysian construction industry, as well as to highlight the importance of construction safety management. The industry has contributed significantly to the economic growth of the country. However, when construction safety management is not implemented systematically, accidents will happen and this can affect the economic growth of the country. This study will try to put the safety management in construction project as one of the important elements to project performance and success. The study will focus on construction project in Malaysia. The study will also emphasize on awareness and the importance of safety management in construction project. The data will be collected by doing the questionnaire and a case study. The analysis of the survey will be done by using the Relative Importance Index (RII) and Cronbach's alpha using SPSS software. The scores were then transformed to importance indices based on the formula