Interactional Slingshots: Providing Support Structure to User Interactions in Hybrid Intelligence Systems

The proliferation of artificial intelligence (AI) systems has enabled us to engage more deeply and powerfully with our digital and physical environments, from chatbots to autonomous vehicles to robotic assistive technology. Unfortunately, these state-of-the-art systems often fail in contexts that require human understanding, are never-before-seen, or complex. In such cases, though the AI-only approaches cannot solve the full task, their ability to solve a piece of the task can be combined with human effort to become more robust to handling complexity and uncertainty. A hybrid intelligence system—one that combines human and machine skill sets—can make intelligent systems more operable in real-world settings. In this dissertation, we propose the idea of using interactional slingshots as a means of providing support structure to user interactions in hybrid intelligence systems. Much like how gravitational slingshots provide boosts to spacecraft en route to their final destinations, so do interactional slingshots provide boosts to user interactions en route to solving tasks. Several challenges arise: What does this support structure look like? How much freedom does the user have in their interactions? How is user expertise paired with that of the machine’s? To do this as a tractable socio-technical problem, we explore this idea in the context of data annotation problems, especially in those domains where AI methods fail to solve the overall task. Getting annotated (labeled) data is crucial for successful AI methods, and becomes especially more difficult in domains where AI fails, since problems in such domains require human understanding to fully solve, but also present challenges related to annotator expertise, annotation freedom, and context curation from the data. To explore data annotation problems in this space, we develop techniques and workflows whose interactional slingshot support structure harnesses the user’s interaction with data. First, we explore providing support in the form of nudging non-expert users’ interactions as they annotate text data for the task of creating conversational memory. Second, we add support structure in the form of assisting non-expert users during the annotation process itself for the task of grounding natural language references to objects in 3D point clouds. Finally, we supply support in the form of guiding expert and non-expert users both before and during their annotations for the task of conversational disentanglement across multiple domains. We demonstrate that building hybrid intelligence systems with each of these interactional slingshot support mechanisms—nudging, assisting, and guiding a user’s interaction with data—improves annotation outcomes, such as annotation speed, accuracy, effort level, even when annotators’ expertise and skill levels vary. Thesis Statement: By providing support structure that nudges, assists, and guides user interactions, it is possible to create hybrid intelligence systems that enable more efficient (faster and/or more accurate) data annotation.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/163138/1/sairohit_1.pd

OntoChatGPT Information System: Ontology-Driven Structured Prompts for ChatGPT Meta-Learning

This research presents a comprehensive methodology for utilizing an ontology-driven structured prompts system in interplay with ChatGPT, a widely used large language model (LLM). The study develops formal models, both information and functional, and establishes the methodological foundations for integrating ontology-driven prompts with ChatGPT's meta-learning capabilities. The resulting productive triad comprises the methodological foundations, advanced information technology, and the OntoChatGPT system, which collectively enhance the effectiveness and performance of chatbot systems. The implementation of this technology is demonstrated using the Ukrainian language within the domain of rehabilitation. By applying the proposed methodology, the OntoChatGPT system effectively extracts entities from contexts, classifies them, and generates relevant responses. The study highlights the versatility of the methodology, emphasizing its applicability not only to ChatGPT but also to other chatbot systems based on LLMs, such as Google's Bard utilizing the PaLM 2 LLM. The underlying principles of meta-learning, structured prompts, and ontology-driven information retrieval form the core of the proposed methodology, enabling their adaptation and utilization in various LLM-based systems. This versatile approach opens up new possibilities for NLP and dialogue systems, empowering developers to enhance the performance and functionality of chatbot systems across different domains and languages.Comment: 14 pages, 1 figure. Published. International Journal of Computing, 22(2), 170-183. https://doi.org/10.47839/ijc.22.2.308

Evaluating the impact of electronic voting systems on university mathematics teaching and learning

This thesis presents an evaluation of the impact of the use of Electronic Voting Systems (EVS) on mathematics teaching and learning, based on the research question: What are the views of academic staff on the impact of EVS use on their mathematics teaching; and how has EVS use influenced student engagement and learning approach to mathematics? To answer the question, a descriptive survey of academic staff, and semi-structured interviews with students were conducted; data from these studies were supplemented by classroom observations of EVS use, relevant documentary evidence, and preliminary studies conducted. Survey data was analysed via quantitative techniques; while the annotated interview transcripts were analysed via thematic analysis, and the application of an integrated theoretical framework. The validity, reliability and replicability of both studies were also established. The findings show that feedback is viewed as the single, most beneficial impact of EVS use, as it enables instructors, through formative assessment, to identify student misconceptions, which then helps instructors to focus on the identified problem areas. EVS has also positively impacted student emotion, behaviour, and cognition. EVS use helps focus student attention, enhances participation and interactivity, and enables students to cognitively engage with learning material. The adoption of an integrated theoretical framework helps to characterise, and to reveal qualitative differences in student learning approaches. Also, the use of specific EVS question types tends to induce specific learning approaches in students. Implications of the findings include the need for EVS-using instructors to have clearly defined pedagogical objectives and well-designed questions, and for learners to re-adapt their mathematical ideas in response to EVS feedback. Findings also show the need to incorporate instructional measures that would promote both procedural and conceptual learning approaches in students, and to perhaps rethink the role of calculator usage and guesswork in student approaches to learning. The requirements for technologies that may replace EVS, the need to align assessment with instructional practices, and for instructors to undergo further EVS training and/or form mathematics-specific support group(s) are also highlighted

Constructing Participatory Environments: a Behavioural Model for Design

This thesis proposes the design of cybernetic frameworks that attempt to explore architecture as ecology of interacting systems that move beyond the fixed and finite tendencies of the past towards spatial environments that are adaptive, emotive and behavioural. Environments within this framework are attempts to construct interaction scenarios that enable agency, curiosity and play, forging intimate exchanges that are participatory and evolving over time. Interaction understood as the evolving relationships between things allows a generative and time-based framework to explore space as a model of interfacing that shifts the tendencies of passive occupancy towards an active ecology of interacting agents. The work argued here moves away from known models that reinforce habitual responses within architecture, towards an understanding of adaptive systems that are active agents for communication and exploration. Architecture within the context of this thesis is explored as a medium for spatial interfacing. Design is thus considered as durational, realtime and anticipatory exploring human human, human machine, and machine machine communication. The challenge posed is how designers can construct environments that are shared, enable curiosity, evolve and allow for complex interactions to arise through human and non-human agency. Attention thus is placed on behavioural features that afford conversational rich exchanges between participants and system, participants with other participants and or systems with other systems. This evolving framework demands that design systems have the capacity to participate and enable new forms of communication. Beyond conventional models that are reactive in their definition of interaction, architecture here moves towards features that are life-like, machine learned, and emotively communicated. The thesis demonstrates and articulates concepts of participation and behaviour through authored prototypes and real-time experiments. Behaviour is not relegated to a generative process in the design phase; rather it is time-based and conversational constantly constructing models of and for communication

Open, Online, Calculus Help Forums: Learning About and From a Public Conversation

This study is an exploration of participation, community, and mathematical understanding in an open, online, calculus help forum. These forums, populated by members from around the world, are locations where students post queries from their coursework and receive assistance from volunteer tutors. The site under investigation has a spontaneous participation structure, meaning that any forum member can respond to a query and contribute to an ongoing discussion. From earlier work, we know that such forums foster mathematical dialogue, contain exchanges with sophisticated pedagogical moves, and exhibit a strong sense of community. In this study, we delve deeper into the functional aspects of activity (such as student positioning and pedagogical moves), the benefits that accrue from participation in tutoring as a communal activity, and the mathematical understanding that is evident in the way problems on limit and related rates are framed and solutions constructed. Based on an observational methodology, we find that the forum provides tutoring for students and support for tutors that is unique from our expectations of other learning environments, such as one-on-one tutoring and computer-based tutoring systems. Students position themselves with authority in the exchanges by making assertions and proposals of action, questioning or challenging others' proposals, and indicating when resolution has been achieved. Tutors, who generally have more experience and expertise than students, provide mathematical guidance, and, in exemplary exchanges, draw the student into making a mathematical discovery. The dedication of tutors to the forum community was evident in the presence of authentic, honest mathematical practices, in the generous provision of alternative perspectives on problems, and in the sincere correction of errors. Some student participants picked up on these aspects of community and expressed excitement and appreciation for this taste of mathematical discourse. The primary contribution of the tutors was their assistance in supporting students as they constructed productive framings for the exercises, and this was the help that students were most in need of. As a result of eavesdropping on this public conversation, we conclude that the forums are a public conversation that should be listened to by educational researchers, teachers, and designers of tutoring systems

Microfoundations of dynamic capabilities in retail in the age of artificial intelligence and robotics

Abstract. The purpose of this study is to find out what are the skills possessed by people working in physical stores and how these can be utilized and managed in the future. The research problem lies in an assumption that do people working in retail stores obtain required skills and capabilities or not, and are they recognized and utilized by management. Many technological solutions have been introduced within the retail industry, but there seems to be lack of understanding how to utilize peoples’ skills differently as some predictable and repeatable tasks are (or will be) performed by machines. There have been studies conducted in the previous years about dynamic capabilities from the perspective of strategic leaders of companies. Dynamic capabilities have been researched in customer centric industries also from the perspective of corporate directors. However, there lacks a study that fully concentrates on retail industry (no other customer centric industries) and solely limits the research to study physical stores mainly from the perspective of people who actually work in the stores, not the leaders in headquarters. Therefore, it could be stated that this study fulfils a research gap lying in the analyses of dynamic capabilities and their microfoundations. The main results of the study show, that there are variations related to information flows as well as there are inefficiencies about information of technological developments happening in the field in general. There is high motivation towards personal and professional development and willingness to learn. Changes made in processes and operations are mostly based on stores’ own customer analyses and communication with their customers. Even though needs for changes are recognized and there is willingness to development, it is highly resources-dependent. Realignments or redeployments are not fully conducted within store operations if their current resources and technologies in use do not allow this. There is a need to move away from certain routines towards being more creative and analytical. Even though there were need to move away from certain routines, there were no indications of considering the routines to be disappearing from store operations. There are structures in stores and in their supportive operations, which enable good customer service, doing the tasks and routines but there are little enabling factors for other things e.g. to innovate and be creative. Skills such as ability to analyse customer behaviour, product knowledge, good communication skills, ability to learn and share new knowledge, plan, organize and prioritize are possessed. However, the structures and processes prevent to utilize these skills to truly fluently communicate new opportunities forward and be part of creating something completely new. Utilization of skills in a new untraditional way are remained in the background because of routines and the traditional role descriptions and therefore it could be argued, that the microfoundations of retail companies are not dynamic

INTEGRATED MODELING AND MONITORING FOR A HEALTHY AND SUSTAINABLE BUILDING ENVIRONMENT

The transmission of airborne diseases indoors is a significant challenge to public health. Buildings are hotspots for viral transmission, which can result in adverse effects on human health and quality of life, especially considering that individuals spend approximately 87% of their time indoors. The emergence of the COVID-19 pandemic has highlighted the importance of considering health aspects during the development of sustainable built environments. Consequently, maintaining a healthy, sustainable, and comfortable built environment represents a major challenge for facilities management teams. However, research on the infection risks associated with emerging pandemics is still in its infancy, and the effectiveness of intervention strategies remains uncertain. Furthermore, the complex interplay between health, energy consumption, and human comfort remains poorly understood, impeding the development of comprehensive control strategies that encompass all three critical dimensions of building sustainability. In addition, existing technologies have limitations to conduct real-time monitoring, while current communication methods between occupants and facilities management teams suffer from a lack of effectiveness, user-friendliness, and informativeness. These deficiencies hinder their ability to address the pressing needs of occupants during pandemics. To address these challenges, this dissertation proposes a convergent framework that integrates modeling, simulation, and monitoring methodologies for the development and maintenance of a sustainable built environment. Airborne transmission risks were first modeled and estimated under different epidemic scenarios, allowing for the evaluation of various intervention strategies. Facility data was then used to develop methods for modeling and simulating the dimensions of energy consumption and thermal comfort, allowing for the identification of tradeoff relationships among health, energy, and comfort, and quantitatively analyzing the impact of indoor environments through HVAC control strategies on the three major dimensions. Finally, an integrated platform was developed to enable the real-time assessment of health, energy, and comfort, including monitoring, visualization, and conversational communication functionalities. The developed framework thus encompasses modeling, simulation, monitoring, and communication capabilities and can be widely adopted by facility management teams, providing insights and guidance to governments and policymakers based on their specific needs. The applicability of the framework extends beyond specific pandemics and can be used to address a broader range of infectious diseases