Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies

Objectives: Machine learning (ML) and natural language processing have great potential to improve effciency and accuracy in diagnosis, treatment recommendations, predictive interventions, and scarce resource allocation within psychiatry. Researchers often conceptualize such an approach as operating in isolation without much need for human involvement, yet it remains crucial to harness human-inthe-loop practices when developing and implementing such techniques as their absence may be catastrophic. We advocate for building ML-based technologies that collaborate with experts within psychiatry in all stages of implementation and use to increase model performance while simultaneously increasing the practicality, robustness, and reliability of the process. Methods: We showcase pitfalls of the traditional ML framework and explain how it can be improved with human-inthe-loop techniques. Specifcally, we applied active learning strategies to the automatic scoring of a story recall task and compared the results to a traditional approach. Results: Human-in-the-loop methodologies supplied a greater understanding of where the model was least confdent or had knowledge gaps during training. As compared to the traditional framework, less than half of the training data were needed to reach a given accuracy. Conclusions: Human-in-the-loop ML is an approach to data collection and model creation that harnesses active learning to select the most critical data needed to increase a model’s accuracy and generalizability more effciently than classic random sampling would otherwise allow. Such techniques may additionally operate as safeguards from spurious predictions and can aid in decreasing disparities that artifcial intelligence systems otherwise propagate

Generative AI and Its Educational Implications

We discuss the implications of generative AI on education across four critical sections: the historical development of AI in education, its contemporary applications in learning, societal repercussions, and strategic recommendations for researchers. We propose ways in which generative AI can transform the educational landscape, primarily via its ability to conduct assessment of complex cognitive performances and create personalized content. We also address the challenges of effective educational tool deployment, data bias, design transparency, and accurate output verification. Acknowledging the societal impact, we emphasize the need for updating curricula, redefining communicative trust, and adjusting to transformed social norms. We end by outlining the ways in which educational stakeholders can actively engage with generative AI, develop fluency with its capacities and limitations, and apply these insights to steer educational practices in a rapidly advancing digital landscape.Comment: This is a preprint version of an edited book chapter to appear in Kourkoulou, D., O. Tzirides, B. Cope, M. Kalantzis, (eds) (2024). Trust and Inclusion in AI-Mediated Education: Where Human Learning Meets Learning Machines, Springe

Thoughts about disordered thinking: measuring and quantifying the laws of order and disorder

Peer ReviewedPostprint (author's final draft

A Synthesis of Post-Fire Road Treatments for BAER Teams: Methods, Treatment Effectiveness, and Decisionmaking Tools for Rehabilitation

We synthesized post-fire road treatment information to assist BAER specialists in making road rehabilitation decisions. We developed a questionnaire; conducted 30 interviews of BAER team engineers and hydrologists; acquired and analyzed gray literature and other relevant publications; and reviewed road rehabilitation procedures and analysis tools. Post-fire road treatments are implemented if the values at risk warrant the treatment and based on regional characteristics, including the timing of first damaging storm and window of implementation. Post-fire peak flow estimation is important when selecting road treatments. Interview results indicate that USGS methods are used for larger watersheds (\u3e5 mi2) and NRCS Curve Number methods are used for smaller watersheds (\u3c5 mi2). These methods are not parameterized and validated for post-fire conditions. Many BAER team members used their own rules to determine parameter values for USGS regression and NRCS CN methods; therefore, there is no consistent way to estimate postfire peak flow. Many BAER road treatments for individual stream crossings were prescribed based on road/culvert surveys, without considering capacities of existing road structure and increased post-fire peak flow. For all regions, rolling dips/water bars, culvert upgrading, and ditch cleaning/armoring are the most frequently used road treatments. For Forest Service Regions 1 and 4, culvert upgrading is preferred, especially for fish-bearing streams. For Forest Service Region 3, culvert removal with temporary road closure and warning signs is preferred. Except for culverts, insufficient data is available on other road treatments to estimate their capacity and to evaluate their effectiveness

Reasoning from multiple texts: An automatic analysis of readers' situation models

In reading multiple texts, a reader must integrate information from the texts with his or her background knowledge. The resulting situation model represents a rich elaborated structure of events, actions, objects, and people involved in the text organized in a manner consistent with the reader's knowledge. In order to evaluate a reader's situation model, a reader's summary must be analyzed in relation to texts the subject has read as well as to more general knowledge such as an expert's knowledge. However, this analysis can be both time-consuming and difficult. In this paper, we use an automatic approach called Latent Semantic Analysis (LSA) for evaluating the situation model of readers of multiple documents. LSA is a statistical model of word usage that generates a high-dimensional semantic space that models the semantics of the text. This paper describes three experiments. The first two describe methods for analyzing a subject's essay to determine from what text a subject learned the information and for grading the quality of information cited in the essay. The third experiment analyzes the knowledge structures of novice and expert readers and compares them to the knowledge structures generated by the model. The experiments illustrate a general approach to modeling and evaluating readers' situation models

La Boheme

Kemp Recital Hall Sunday and Monday Evening March 31 and April 1, 1996 8.00 p.m

Extending the usefulness of the verbal memory test: The promise of machine learning

The evaluation of verbal memory is a core component of neuropsychological assessment in a wide range of clinical and research settings. Leveraging story recall to assay neurocognitive function could be made more useful if it were possible to administer frequently (i.e., would allow for the collection of more patient data over time) and automatically assess the recalls with machine learning methods. In the present study, we evaluated a novel story recall test with 24 parallel forms that was deployed using smart devices in 94 psychiatric inpatients and 80 nonpatient adults. Machine learning and vector-based natural language processing methods were employed to automate test scoring, and performance using these methods was evaluated in their incremental validity, criterion validity (i.e., convergence with trained human raters), and parallel forms reliability. Our results suggest moderate to high consistency across the parallel forms, high convergence with human raters (r values ~ 0.89), and high incremental validity for discriminating between groups. While much work remains, the present findings are critical for implementing an automated, neuropsychological test deployable using remote technologies across multiple and frequent administrations

Towards a temporospatial framework for measurements of disorganization in speech using semantic vectors

Incoherent speech in schizophrenia has long been described as the mind making “leaps” of large distances between thoughts and ideas. Such a view seems intuitive, and for almost two decades, attempts to operationalize these conceptual “leaps” in spoken word meanings have used language-based embedding spaces. An embedding space represents meaning of words as numerical vectors where a greater proximity between word vectors represents more shared meaning. However, there are limitations with word vector-based operationalizations of coherence which can limit their appeal and utility in clinical practice. First, the use of esoteric word embeddings can be conceptually hard to grasp, and this is complicated by several different operationalizations of incoherent speech. This problem can be overcome by a better visualization of methods. Second, temporal information from the act of speaking has been largely neglected since models have been built using written text, yet speech is spoken in real time. This issue can be resolved by leveraging time stamped transcripts of speech. Third, contextual information - namely the situation of where something is spoken - has often only been inferred and never explicitly modeled. Addressing this situational issue opens up new possibilities for models with increased temporal resolution and contextual relevance. In this paper, direct visualizations of semantic distances are used to enable the inspection of examples of incoherent speech. Some common operationalizations of incoherence are illustrated, and suggestions are made for how temporal and spatial contextual information can be integrated in future implementations of measures of incoherence

Benefits and Challenges of Multidisciplinary Project Teams: Lessons Learned for Researchers and Practitioners

Adopting a multidisciplinary research approach would enable test and evaluation professionals to more effective!y investigate the complex human performance problems faced in today\u27s technologically advanced operational domains. To illustrate the utility of this approach, we present lessons learned based on our experiences as a multi-agency, multidisciplinary team collaborating on an Army research project involving a dynamic military command and control simulation. Our goal with these lessons learned is to provide guidance to researchers and practitioners alike concerning the benefits and challenges of such collaboration. Our project team\u27s diverse members, drawn from both industry and government organizations, offer their multiple p perspectives on these issues. The final sections then summarize the challenges and benefits of multidisciplinary research