Item Selection Methods Based on Multiple Objective Approaches for Classifying Respondents Into Multiple Levels

Computerized classification tests classify examinees into two or more levels while maximizing accuracy and minimizing test length. The majority of currently available item selection methods maximize information at one point on the ability scale, but in a test with multiple cutting points selection methods could take all these points simultaneously into account. If for each cutting point one objective is specified, the objectives can be combined into one optimization function using multiple objective approaches. Simulation studies were used to compare the efficiency and accuracy of eight selection methods in a test based on the sequential probability ratio test. Small differences were found in accuracy and efficiency between different methods depending on the item pool and settings of the classification method. The size of the indifference region had little influence on accuracy but considerable influence on efficiency. Content and exposure control had little influence on accuracy and efficienc

Theoretical and Practical Advances in Computer-based Educational Measurement

This open access book presents a large number of innovations in the world of operational testing. It brings together different but related areas and provides insight in their possibilities, their advantages and drawbacks. The book not only addresses improvements in the quality of educational measurement, innovations in (inter)national large scale assessments, but also several advances in psychometrics and improvements in computerized adaptive testing, and it also offers examples on the impact of new technology in assessment. Due to its nature, the book will appeal to a broad audience within the educational measurement community. It contributes to both theoretical knowledge and also pays attention to practical implementation of innovations in testing technology

Online Assessment in Large Undergraduate Courses During COVID-19 Emergency Response Teaching

The transition to online instruction during the COVID-19 pandemic was unprecedented and forced many universities to quickly embrace online distance learning. This context created new challenges, particularly around assessment strategies. Empirical research has demonstrated that formative assessment fosters more active learning in online classrooms. However, formative assessment strategies are not always adapted well to online platforms based on the nature of the subject matter and the size of the class. This qualitative case study sought to understand instructors’ experiences and strategies for conducting assessment remotely, specifically for large-size undergraduate courses. The investigation relied on data from semi-structured interviews with University of Maryland, College Park instructors who received a Teaching Innovation Grant from the Provost’s Office in Summer 2020 intended to fund sustainable online delivery beyond the emergency response teaching phase. For this analysis, we analyzed the transcripts of 13 interviews, representing a diverse range of programs, schools, and faculty seniority levels at the university. Findings show instructors experienced several successes during course retooling, including significant increases in student performance. Most instructors also indicated that they would continue to keep new online assessment strategies for the future, regardless of whether that future includes online, blended, or in-person delivery. Despite the anticipation that the pandemic would fuel more opportunities for cheating, there was only one experience of academic dishonesty

A novel approach to neutron dosimetry

Purpose: Having been overlooked for many years, research is now starting to take into account the directional distribution of neutron workplace fields. Existing neutron dosimetry instrumentation does not account for this directional distribution, resulting in conservative estimates of dose in neutron workplace fields (by around a factor of 2, although this is heavily dependent on the type of field). This conservatism could influence epidemiological studies on the health effects of radiation exposure. This paper reports on the development of an instrument which can estimate the effective dose of a neutron field, accounting for both the direction and the energy distribution. Methods: A 6Li-loaded scintillator was used to perform neutron assays at a number of locations in a 20 × 20 × 17.5 cm3 water phantom. The variation in thermal and fast neutron response to different energies and field directions was exploited. The modeled response of the instrument to various neutron fields was used to train an artificial neural network (ANN) to learn the effective dose and ambient dose equivalent of these fields. All experimental data published in this work were measured at the National Physical Laboratory (UK). Results: Experimental results were obtained for a number of radionuclide source based neutron fields to test the performance of the system. The results of experimental neutron assays at 25 locations in a water phantom were fed into the trained ANN. A correlation between neutron counting rates in the phantom and neutron fluence rates was experimentally found to provide dose rate estimates. A radionuclide source behind shadow cone was used to create a more complex field in terms of energy and direction. For all fields, the resulting estimates of effective dose rate were within 45% or better of their calculated values, regardless of energy distribution or direction for measurement times greater than 25 min. Conclusions: This work presents a novel, real-time, approach to workplace neutron dosimetry. It is believed that in the research presented in this paper, for the first time, a single instrument has been able to estimate effective dose

Perfectionism, Negative Affect, Anxiety, and Self-evaluations for Brief Tasks

Perfectionism is a trait with multiple dimensions, which vary in terms of associated costs and benefits. Maladaptive perfectionism is related to neuroticism and involves self-criticism and perceptions of difficulty meeting high standards. In contrast, adaptive perfectionism is associated with conscientiousness and can be considered the healthy pursuit of high standards with minimal distress. Assessment of perfectionism has primarily been limited to self-report, so the present study investigated relationships between perfectionism dimensions and responses to a computerized search task in a sample of 133 undergraduates. In addition, friends and parents were asked to rate several traits of the participant using an online survey. The cost of errors for the task was manipulated, and maladaptive perfectionism subscales were hypothesized to predict worse performance and more task-related distress. Although neither maladaptive perfectionism nor adaptive perfectionism predicted task performance as hypothesized, maladaptive perfectionism predicted worse reactions: e.g., activated negative affect, frustration) to the task; however, incremental validity was limited. Unexpectedly, post-hoc analyses revealed that adaptive perfectionism predicted more frustration and less satisfaction for the task above and beyond conscientiousness. Informant ratings of participant personality traits demonstrated agreement, even for less observable measures, and achieved incremental validity beyond similar participant ratings in a few instances: e.g., task confidence). Informant ratings of personality seem to be useful supplements to self-report perfectionism measures. In addition, a brief task may not be suitable for observing the distinctive behavioral patterns of perfectionists. Although perfectionism dimensions overlap considerably with higher order personality constructs, they can provide unique information about meaningful outcomes. Recommendations for future research and implications are discussed

Perspectives and Best Practices for Artificial Intelligence and Continuously Learning Systems in Healthcare

Goals of this paper Healthcare is often a late adopter when it comes to new techniques and technologies; this works to our advantage in the development of this paper as we relied on lessons learned from CLS in other industries to help guide the content of this paper. Appendix V includes a number of example use cases of AI in Healthcare and other industries. This paper focuses on identifying unique attributes, constraints and potential best practices towards what might represent “good” development for Continuously Learning Systems (CLS) AI systems with applications ranging from pharmaceutical applications for new drug development and research to AI enabled smart medical devices. It should be noted that although the emphasis of this paper is on CLS, some of these issues are common to all AI products in healthcare. Additionally, there are certain topics that should be considered when developing CLS for healthcare, but they are outside of the scope of this paper. These topics will be briefly touched upon, but will not be explored in depth. Some examples include: Human Factors – this is a concern in the development of any product – what are the unique usability challenges that arise when collecting data and presenting the results? Previous efforts at generating automated alerts have often created problems (e.g. alert fatigue.) CyberSecurity and Privacy – holding a massive amount of patient data is an attractive target for hackers, what steps should be taken to protect data from misuse? How does the European Union’s General Data Protection Regulation (GDPR) impact the use of patient data? Legal liability – if a CLS system recommends action that is then reviewed and approved by a doctor, where does the liability lie if the patient is negatively affected? Regulatory considerations – medical devices are subject to regulatory oversight around the world; in fact, if a product is considered a medical device depends on what country you are in. AI provides an interesting challenge to traditional regulatory models. Additionally, some organizations like the FTC regulate non-medical devices. This paper is not intended to be a standard, nor is this paper trying to advocate for one and only one method of developing, verifying, and validating CLS systems – this paper highlights best practices from other industries and suggests adaptation of those processes for healthcare. This paper is also not intended to evaluate existing or developing regulatory, legal, ethical, or social consequences of CLS systems. This is a rapidly evolving subject with many companies, and now some countries, establishing their own AI Principles or Code of Conduct which emphasize legal and ethical considerations including goals and principles of fairness, reliability and safety, transparency around how the results of these learning systems are explained to the people using those systems5 . The intended audience of this paper are Developers, Researchers, Quality Assurance and Validation personnel, Business Managers and Regulators across both Medical Device and Pharmaceutical industries that would like to learn more about CLS best practices, and CLS practitioners wanting to learn more about medical device software development

Conceptual Design Model of Computerized Personal-Decision AID (ComPDA)

To date, the attentions given to the improvement of decision support at organizational level has been enormous. On the contrary, academic research in improving the performance of computerized decision aid (CDA) for personal decision is lacking, in which some are dated. Nowadays, the existence of CDA which handles personal decision is mushrooming and progressively getting attention from users. Despite that, users’ perceptions of the suitable decision strategy and technique for CDA have not been subjected to systematic investigation. Literature reviews also indicate that most users do not go for complex mathematical techniques despite the fact that these techniques are better at handling the risks and uncertainties in decisions. In fact, more often than not, the development process of CDAs does not seem to adhere to any conceptual and theoretical model. In view of that, this study aims to propose a conceptual design model for computerized personal-decision aid (ComPDA). The following objectives are outlined to support the general aim: (i) to identify appropriate decision strategy and technique for ComPDA, (ii) to incorporate identified strategy and technique in the construction of conceptual design model for ComPDA (iii) to validate the conceptual design model in different situations via prototyping method and (iv) to measure the users’ perceived helpfulness of the ComPDA prototypes. Participatory design method was implemented in order to achieve objective i and ii. The findings were incorporated into the construction of the conceptual design model of ComPDA. In achieving objective iii, the conceptual design model was validated in two different case studies via prototyping: A- choosing development methodology in mobile computing course (md-Matrix); and B- purchasing a mobile phone (ep-Matrix). In achieving objective iv, an instrument (named as Q-HELP) was developed to measure the helpfulness (HLP) of the prototypes. This study identified four relevant constructs pertinent to helpfulness; reliability (REL), decision making effort (EFF), confidence (CON), and decision awareness (AWR). Altogether, 122 respondents participated where 63 were from case study A and 59 from case study B. Eight hypotheses were formulated comprising testing for correlation between all the constructs in Q-HELP with helpfulness, testing the average time spent to make a selection with and without the proposed ComPDA and testing if the mean score of helpfulness of the proposed ComPDA is high. Paired Samples t Test, Pearson Correlation analyses and descriptive analyses were utilized to validate the hypotheses. The results show that: REL and HLP are significantly correlated, EFF and HLP are significantly correlated, CON and HLP are significantly correlated, AWR and HLP are significantly correlated, the use of md-Matrix and ep-Matrix significantly reduces the time spent to make selection, mean score of helpfulness of md-Matrix is fairly high and mean score of helpfulness of ep-Matrix is high. However, it is concluded that the overall results exhibit sufficient indication that md-Matrix and ep-Matrix were found helpful to users in terms of reliability, lessening the decision making effort, increasing confidence and also awareness in decision making. This study has produced the following outcomes, along with achieving all of its objectives: (i) a conceptual design model for ComPDA which incorporates suitable decision strategies and techniques identified via systematic investigations; (ii) two functional ComPDA prototypes to validate the conceptual design model and to demonstrate its applicability in different situations, (iii) an instrument for measuring helpfulness which includes dimensions from outcome and process aspects; and (iv) comparative analyses of decision models, strategies and techniques which provide basis for future studies.

A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment

Research has shown that aggregation of independent expert judgments significantly improves the quality of forecasts as compared to individual expert forecasts. This “wisdom of crowds” (WOC) has sparked substantial interest. However, previous studies on strengths and weaknesses of aggregation algorithms have been restricted by limited empirical data and analytical complexity. Based on a comprehensive analysis of existing knowledge on WOC and aggregation algorithms, this paper describes the design and implementation of a static stochastic simulation model to emulate WOC scenarios with a wide range of parameters. The model has been thoroughly evaluated: the assumptions are validated against propositions derived from literature, and the model has a computational representation. The applicability of the model is demonstrated by investigating aggregation algorithm behavior on a detailed level, by assessing aggregation algorithm performance, and by exploring previously undiscovered suppositions on WOC. The simulation model helps expand the understanding of WOC, where previous research was restricted. Additionally, it gives directions for developing aggregation algorithms and contributes to a general understanding of the WOC phenomenon