TOWARDS BUILDING INTELLIGENT COLLABORATIVE PROBLEM SOLVING SYSTEMS

Historically, Collaborative Problem Solving (CPS) systems were more focused on Human Computer Interaction (HCI) issues, such as providing good experience of communication among the participants. Whereas, Intelligent Tutoring Systems (ITS) focus both on HCI issues as well as leveraging Artificial Intelligence (AI) techniques in their intelligent agents. This dissertation seeks to minimize the gap between CPS systems and ITS by adopting the methods used in ITS researches. To move towards this goal, we focus on analyzing interactions with textual inputs in online learning systems such as DeepTutor and Virtual Internships (VI) to understand their semantics and underlying intents. In order to address the problem of assessing the student generated short text, this research explores firstly data driven machine learning models coupled with expert generated as well as general text analysis features. Secondly it explores method to utilize knowledge graph embedding for assessing student answer in ITS. Finally, it also explores a method using only standard reference examples generated by human teacher. Such method is useful when a new system has been deployed and no student data were available.To handle negation in tutorial dialogue, this research explored a Long Short Term Memory (LSTM) based method. The advantage of this method is that it requires no human engineered features and performs comparably well with other models using human engineered features.Another important analysis done in this research is to find speech acts in conversation utterances of multiple players in VI. Among various models, a noise label trained neural network model performed better in categorizing the speech acts of the utterances.The learners\u27 professional skill development in VI is characterized by the distribution of SKIVE elements, the components of epistemic frames. Inferring the population distribution of these elements could help to assess the learners\u27 skill development. This research sought a Markov method to infer the population distribution of SKIVE elements, namely the stationary distribution of the elements.While studying various aspects of interactions in our targeted learning systems, we motivate our research to replace the human mentor or tutor with intelligent agent. Introducing intelligent agent in place of human helps to reduce the cost as well as scale up the system

by

th

Peer Attention Modeling with Head Pose Trajectory Tracking Using Temporal Thermal Maps

Human head pose trajectories can represent a wealth of implicit information such as areas of attention, body language, potential future actions, and more. This signal is of high value for use in Human-Robot teams due to the implicit information encoded within it. Although team-based tasks require both explicit and implicit communication among peers, large team sizes, noisy environments, distance, and mission urgency can inhibit the frequency and quality of explicit communication. The goal for this thesis is to improve the capabilities of Human-Robot teams by making use of implicit communication. In support of this goal, the following hypotheses are investigated: ● Implicit information about a human subject’s attention can be reliably extracted with software by tracking the subject’s head pose trajectory, and ● Attention can be represented with a 3D temporal thermal map for implicitly determining a subject’s Objects Of Interest (OOIs). These hypotheses are investigated by experimentation with a new tool for peer attention modeling by Head Pose Trajectory Tracking using Temporal Thermal Maps (HPT4M). This system allows a robot Observing Agent (OA) to view a human teammate and temporally model their Regions Of Interest (ROIs) by generating a 3D thermal map based on the subject’s head pose trajectory. The findings in this work are that HPT4M can be used by an OA to contribute to a team search mission by implicitly discovering a human subject’s OOI type, mapping the item’s location within the searched space, and labeling the item’s discovery state. Furthermore, this work discusses some of the discovered limitations of this technology and hurdles that must be overcome before implementing HPT4M in a reliable real-world system. Finally, the techniques used in this work are provided as an open source Robot Operating System (ROS) node at github.com/HPT4M with the intent that it will aid other developers in the robotics community with improving Human-Robot teams. Furthermore, the proofs of principle and tools developed in this thesis are a foundational platform for deeper investigation in future research on improving Human-Robot teams via implicit communication techniques

Mining Behavior of Citizen Sensor Communities to Improve Cooperation with Organizational Actors

Web 2.0 (social media) provides a natural platform for dynamic emergence of citizen (as) sensor communities, where the citizens generate content for sharing information and engaging in discussions. Such a citizen sensor community (CSC) has stated or implied goals that are helpful in the work of formal organizations, such as an emergency management unit, for prioritizing their response needs. This research addresses questions related to design of a cooperative system of organizations and citizens in CSC. Prior research by social scientists in a limited offline and online environment has provided a foundation for research on cooperative behavior challenges, including \u27articulation\u27 and \u27awareness\u27, but Web 2.0 supported CSC offers new challenges as well as opportunities. A CSC presents information overload for the organizational actors, especially in finding reliable information providers (for awareness), and finding actionable information from the data generated by citizens (for articulation). Also, we note three data level challenges: ambiguity in interpreting unconstrained natural language text, sparsity of user behaviors, and diversity of user demographics. Interdisciplinary research involving social and computer sciences is essential to address these socio-technical issues. I present a novel web information-processing framework, called the Identify-Match- Engage (IME) framework. IME allows operationalizing computation in design problems of awareness and articulation of the cooperative system between citizens and organizations, by addressing data problems of group engagement modeling and intent mining. The IME framework includes: a.) Identification of cooperation-assistive intent (seeking-offering) from short, unstructured messages using a classification model with declarative, social and contrast pattern knowledge, b.) Facilitation of coordination modeling using bipartite matching of complementary intent (seeking-offering), and c.) Identification of user groups to prioritize for engagement by defining a content-driven measure of \u27group discussion divergence\u27. The use of prior knowledge and interplay of features of users, content, and network structures efficiently captures context for computing cooperation-assistive behavior (intent and engagement) from unstructured social data in the online socio-technical systems. Our evaluation of a use-case of the crisis response domain shows improvement in performance for both intent classification and group engagement prioritization. Real world applications of this work include use of the engagement interface tool during various recent crises including the 2014 Jammu and Kashmir floods, and intent classification as a service integrated by the crisis mapping pioneer Ushahidi\u27s CrisisNET project for broader impact

Personalized face and gesture analysis using hierarchical neural networks

The video-based computational analyses of human face and gesture signals encompass a myriad of challenging research problems involving computer vision, machine learning and human computer interaction. In this thesis, we focus on the following challenges: a) the classification of hand and body gestures along with the temporal localization of their occurrence in a continuous stream, b) the recognition of facial expressivity levels in people with Parkinson's Disease using multimodal feature representations, c) the prediction of student learning outcomes in intelligent tutoring systems using affect signals, and d) the personalization of machine learning models, which can adapt to subject and group-specific nuances in facial and gestural behavior. Specifically, we first conduct a quantitative comparison of two approaches to the problem of segmenting and classifying gestures on two benchmark gesture datasets: a method that simultaneously segments and classifies gestures versus a cascaded method that performs the tasks sequentially. Second, we introduce a framework that computationally predicts an accurate score for facial expressivity and validate it on a dataset of interview videos of people with Parkinson's disease. Third, based on a unique dataset of videos of students interacting with MathSpring, an intelligent tutoring system, collected by our collaborative research team, we build models to predict learning outcomes from their facial affect signals. Finally, we propose a novel solution to a relatively unexplored area in automatic face and gesture analysis research: personalization of models to individuals and groups. We develop hierarchical Bayesian neural networks to overcome the challenges posed by group or subject-specific variations in face and gesture signals. We successfully validate our formulation on the problems of personalized subject-specific gesture classification, context-specific facial expressivity recognition and student-specific learning outcome prediction. We demonstrate the flexibility of our hierarchical framework by validating the utility of both fully connected and recurrent neural architectures

Assessing and predicting the students’ systems thinking preference: multi-criteria decision making and machine learning

The 21st century is marked by a technological revolution that features digital implementation and high interconnectivity between systems across different domains, such as transportation, agriculture, education, and health. Although these technological changes resulted in modern systems capable of easing individuals’ lives, these systems are increasingly complex, and that increased complexity is only expected to continue. The increased system complexity is due to the rapid exchange of information between subsystems, which creates high interconnectivity and interdependence between the subsystems and their elements. Workforce skill sets, as a result, must be modified appropriately to ensure the systems’ success. Systems Thinking is an approach that helps individuals better understand and effectively solve modern complex systems problems by encouraging holistic thinking. Systems thinking consists of two approaches holistic and reductionist views. This dissertation aims to study college engineering and non-engineering students’ preference for holistic thinking versus reductionist thinking, their ranking to the systems thinking dimensions, and whether this preference varies depending on demographics and general factors. Additionally, this study investigates the possibility of predicting the students’ preference for holistic thinking. The study uses the multi-criteria decision-making method, the Analytic Hierarchy Process and Fuzzy Analytic Hierarchy Process to determine the student’s preferences, and uses statistical analysis such as independent sample t-test and ANOVA to evaluate the factors. Also, the study uses machine learning classification models such as Logistic Regression, Support Vector Machine, Naïve Bayes, Decision Trees, voting classifiers, Bagging, and Random Forest to predict and evaluate the most predicting model. The results of the dissertation conclude that overall students prefer the reductionist approach and report the students’ preference towards dimensions of complexity, independence, uncertainty, systems worldview, and flexibility and the ranking difference based on some factors. Lastly, the results show that the students’ preference for holistic thinking can be predicted with a 77% accuracy using the Random Forest classifier

Systems for pervasive electronics and interfaces

Energy Harvesting Active Networked Tags (EnHANTs) are a new type of wireless device in the domain between RFIDs and sensor networks. Future EnHANTs will be small, flexible, and self-powered devices that can be attached to everyday objects that are traditionally not networked to enable "Internet of Things" applications. This work describes the design and development of the EnHANT prototypes and testbed. The current prototypes use thin-film photovoltaics optimized for indoor light harvesting, form multihop networks using ultra-low-power Ultra-Wideband Impulse Radio (UWB-IR) transceivers, and implement energy harvesting adaptive networking protocols. The current testbed enables the evaluation of different algorithms by exposing individual prototypes to repeatable light conditions based on real-world irradiance data. New approaches to characterizing the energy available to energy harvesting devices were explored. A mobile data-logger was used to record the intensity of ambient light, determine the light source, and record the acceleration from motion during different real world activities. These traces were used to model the behavior of photovoltaic and inertial energy harvesters in real world deployments and can be replayed in the EnHANTs testbed. In addition, new techniques to evaluate the efficiency of different photovoltaic technologies under indoor illumination were developed. A proof-of-concept system was built to characterize photovoltaics under a standardized set of conditions in which the radiant intensity and spectral composition of the light source were systematically varied. Techniques to structure student research projects within the EnHANTs project were developed. Project-based learning approaches were implemented to engage students using real-world system development constraints. A survey of the students showed that this approach is an effective method for developing technical, professional, and soft skills. Open source hardware was also applied to EnHANTs project and extended into other domains. A laboratory-based class in flat panel display technology was developed. The course introduces fundamental concepts of display systems and reinforces these concepts through the fabrication of three display devices. A lab kit platform was developed to enable remote students to use low-cost, course specific hardware to complete the lab exercises remotely. This platform was also applied to external projects targeted at non-university students. A workshop was developed to teach artists, designers, and hobbyists how to design and build custom user interfaces using thin-film electronics and rapid prototyping tools. Surveys of the students and workshop participants showed that this platform is an effective teaching tool and can be easily adapted and expanded