Evaluating true BCI communication rate through mutual information and language models.

Brain-computer interface (BCI) systems are a promising means for restoring communication to patients suffering from "locked-in" syndrome. Research to improve system performance primarily focuses on means to overcome the low signal to noise ratio of electroencephalogric (EEG) recordings. However, the literature and methods are difficult to compare due to the array of evaluation metrics and assumptions underlying them, including that: 1) all characters are equally probable, 2) character selection is memoryless, and 3) errors occur completely at random. The standardization of evaluation metrics that more accurately reflect the amount of information contained in BCI language output is critical to make progress. We present a mutual information-based metric that incorporates prior information and a model of systematic errors. The parameters of a system used in one study were re-optimized, showing that the metric used in optimization significantly affects the parameter values chosen and the resulting system performance. The results of 11 BCI communication studies were then evaluated using different metrics, including those previously used in BCI literature and the newly advocated metric. Six studies' results varied based on the metric used for evaluation and the proposed metric produced results that differed from those originally published in two of the studies. Standardizing metrics to accurately reflect the rate of information transmission is critical to properly evaluate and compare BCI communication systems and advance the field in an unbiased manner

A Prototype Toolkit For Evaluating Indoor Environmental Quality In Commercial Buildings

Measurement of building environmental parameters is often complex, expensive, and not easily proceduralized in a manner that covers all commercial buildings. Evaluating building indoor environmental quality performance is therefore not standard practice. This project developed a prototype toolkit that addressed existing barriers to widespread indoor environmental quality performance evaluation. A toolkit with both hardware and software elements was designed for practitioners around the indoor environmental quality requirements of the American Society of Heating, Refrigeration and Air Conditioning Engineers / Chartered Institution of Building Services / United States Green Building Council Performance Measurement Protocols. This unique toolkit was built on a wireless mesh network with a web-based data collection, analysis, and reporting application. The toolkit provided a fast, robust deployment of sensors, real-time data analysis, Performance Measurement Protocol-based analysis methods and a scorecard and report generation tools. A web-enabled Geographic Information System-based metadata collection system also reduced field-study deployment time. The toolkit was evaluated through three case studies, which were discussed in this report

How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility

Recommendation systems are ubiquitous and impact many domains; they have the potential to influence product consumption, individuals' perceptions of the world, and life-altering decisions. These systems are often evaluated or trained with data from users already exposed to algorithmic recommendations; this creates a pernicious feedback loop. Using simulations, we demonstrate how using data confounded in this way homogenizes user behavior without increasing utility

The Effects of the Quantification of Faculty Productivity: Perspectives from the Design Science Research Community

In recent years, efforts to assess faculty research productivity have focused more on the measurable quantification of academic outcomes. For benchmarking academic performance, researchers have developed different ranking and rating lists that define so-called high-quality research. While many scholars in IS consider lists such as the Senior Scholar’s basket (SSB) to provide good guidance, others who belong to less-mainstream groups in the IS discipline could perceive these lists as constraining. Thus, we analyzed the perceived impact of the SSB on information systems (IS) academics working in design science research (DSR) and, in particular, how it has affected their research behavior. We found the DSR community felt a strong normative influence from the SSB. We conducted a content analysis of the SSB and found evidence that some of its journals have come to accept DSR more. We note the emergence of papers in the SSB that outline the role of theory in DSR and describe DSR methodologies, which indicates that the DSR community has rallied to describe what to expect from a DSR manuscript to the broader IS community and to guide the DSR community on how to organize papers for publication in the SSB

A Classification and Survey of Computer System Performance Evaluation Techniques

Classification and survey of computer system performance evaluation technique

Cognitive apprenticeship : teaching the craft of reading, writing, and mathtematics

Includes bibliographical references (p. 25-27)This research was supported by the National Institute of Education under Contract no. US-NIE-C-400-81-0030 and the Office of Naval Research under Contract No. N00014-85-C-002

Active Learning: Effects of Core Training Design Elements on Self-Regulatory Processes, Learning, and Adaptability

This research describes a comprehensive examination of the cognitive, motivational, and emotional processes underlying active learning approaches, their effects on learning and transfer, and the core training design elements (exploration, training frame, emotion-control) and individual differences (cognitive ability, trait goal orientation, trait anxiety) that shape these processes. Participants (N = 350) were trained to operate a complex computer-based simulation. Exploratory learning and error-encouragement framing had a positive effect on adaptive transfer performance and interacted with cognitive ability and dispositional goal orientation to influence trainees’ metacognition and state goal orientation. Trainees who received the emotion-control strategy had lower levels of state anxiety. Implications for developing an integrated theory of active learning, learner-centered design, and research extensions are discussed