Recognising Complex Mental States from Naturalistic Human-Computer Interactions

New advances in computer vision techniques will revolutionize the way we interact with computers, as they, together with other improvements, will help us build machines that understand us better. The face is the main non-verbal channel for human-human communication and contains valuable information about emotion, mood, and mental state. Affective computing researchers have investigated widely how facial expressions can be used for automatically recognizing affect and mental states. Nowadays, physiological signals can be measured by video-based techniques, which can also be utilised for emotion detection. Physiological signals, are an important indicator of internal feelings, and are more robust against social masking. This thesis focuses on computer vision techniques to detect facial expression and physiological changes for recognizing non-basic and natural emotions during human-computer interaction. It covers all stages of the research process from data acquisition, integration and application. Most previous studies focused on acquiring data from prototypic basic emotions acted out under laboratory conditions. To evaluate the proposed method under more practical conditions, two different scenarios were used for data collection. In the first scenario, a set of controlled stimulus was used to trigger the user’s emotion. The second scenario aimed at capturing more naturalistic emotions that might occur during a writing activity. In the second scenario, the engagement level of the participants with other affective states was the target of the system. For the first time this thesis explores how video-based physiological measures can be used in affect detection. Video-based measuring of physiological signals is a new technique that needs more improvement to be used in practical applications. A machine learning approach is proposed and evaluated to improve the accuracy of heart rate (HR) measurement using an ordinary camera during a naturalistic interaction with computer

Recognising Complex Mental States from Naturalistic Human-Computer Interactions

New advances in computer vision techniques will revolutionize the way we interact with computers, as they, together with other improvements, will help us build machines that understand us better. The face is the main non-verbal channel for human-human communication and contains valuable information about emotion, mood, and mental state. Affective computing researchers have investigated widely how facial expressions can be used for automatically recognizing affect and mental states. Nowadays, physiological signals can be measured by video-based techniques, which can also be utilised for emotion detection. Physiological signals, are an important indicator of internal feelings, and are more robust against social masking. This thesis focuses on computer vision techniques to detect facial expression and physiological changes for recognizing non-basic and natural emotions during human-computer interaction. It covers all stages of the research process from data acquisition, integration and application. Most previous studies focused on acquiring data from prototypic basic emotions acted out under laboratory conditions. To evaluate the proposed method under more practical conditions, two different scenarios were used for data collection. In the first scenario, a set of controlled stimulus was used to trigger the user’s emotion. The second scenario aimed at capturing more naturalistic emotions that might occur during a writing activity. In the second scenario, the engagement level of the participants with other affective states was the target of the system. For the first time this thesis explores how video-based physiological measures can be used in affect detection. Video-based measuring of physiological signals is a new technique that needs more improvement to be used in practical applications. A machine learning approach is proposed and evaluated to improve the accuracy of heart rate (HR) measurement using an ordinary camera during a naturalistic interaction with computer

Determining what people feel and think when interacting with humans and machines

Any interactive software program must interpret the users’ actions and come up with an appropriate response that is intelligable and meaningful to the user. In most situations, the options of the user are determined by the software and hardware and the actions that can be carried out are unambiguous. The machine knows what it should do when the user carries out an action. In most cases, the user knows what he has to do by relying on conventions which he may have learned by having had a look at the instruction manual, having them seen performed by somebody else, or which he learned by modifying a previously learned convention. Some, or most, of the times he just finds out by trial and error. In user-friendly interfaces, the user knows, without having to read extensive manuals, what is expected from him and how he can get the machine to do what he wants. An intelligent interface is so-called, because it does not assume the same kind of programming of the user by the machine, but the machine itself can figure out what the user wants and how he wants it without the user having to take all the trouble of telling it to the machine in the way the machine dictates but being able to do it in his own words. Or perhaps by not using any words at all, as the machine is able to read off the intentions of the user by observing his actions and expressions. Ideally, the machine should be able to determine what the user wants, what he expects, what he hopes will happen, and how he feels

Requirements for Robotic Interpretation of Social Signals “in the Wild”: Insights from Diagnostic Criteria of Autism Spectrum Disorder

The last few decades have seen widespread advances in technological means to characterise observable aspects of human behaviour such as gaze or posture. Among others, these developments have also led to significant advances in social robotics. At the same time, however, social robots are still largely evaluated in idealised or laboratory conditions, and it remains unclear whether the technological progress is sufficient to let such robots move “into the wild”. In this paper, we characterise the problems that a social robot in the real world may face, and review the technological state of the art in terms of addressing these. We do this by considering what it would entail to automate the diagnosis of Autism Spectrum Disorder (ASD). Just as for social robotics, ASD diagnosis fundamentally requires the ability to characterise human behaviour from observable aspects. However, therapists provide clear criteria regarding what to look for. As such, ASD diagnosis is a situation that is both relevant to real-world social robotics and comes with clear metrics. Overall, we demonstrate that even with relatively clear therapist-provided criteria and current technological progress, the need to interpret covert behaviour cannot yet be fully addressed. Our discussions have clear implications for ASD diagnosis, but also for social robotics more generally. For ASD diagnosis, we provide a classification of criteria based on whether or not they depend on covert information and highlight present-day possibilities for supporting therapists in diagnosis through technological means. For social robotics, we highlight the fundamental role of covert behaviour, show that the current state-of-the-art is unable to characterise this, and emphasise that future research should tackle this explicitly in realistic settings

Researching sustainable agriculture: The role of values in systemic science

This paper presents a specific perspective on the science demarcation issue, the perspective of systemic science. A systemic science is a science that influences its own subject area. Agricultural science is an example of such a science - a point that is particularly evident in connection with research in organic farming, which forms the practical context of this paper. Far from the ideal of being 'value-free' and objective, the systemic science must, upon recognising itself as systemic, acknowledge the role of values in research and include value inquiry as a specific research task. But still, the systemic science insists that it is science. Given that it is science, what then demarcates 'science' as different from other social activities? Or, in other terms, what are the proper criteria of scientific quality for systemic sciences? The paper aims to develop the conception of systemic science and investigate some basic aspects of science as a learning process, in order to work towards a more adequate foundation for developing and evaluating systemic research methods, and for determining appropriate criteria of scientific quality in systemic science

Structured evaluation of virtual environments for special-needs education

This paper describes the development of a structured approach to evaluate experiential and communication virtual learning environments (VLEs) designed specifically for use in the education of children with severe learning difficulties at the Shepherd special needs school in Nottingham, UK. Constructivist learning theory was used as a basis for the production of an evaluation framework, used to evaluate the design of three VLEs and how they were used by students with respect to this learning theory. From an observational field study of student-teacher pairs using the VLEs, 18 behaviour categories were identified as relevant to five of the seven constructivist principles defined by Jonassen (1994). Analysis of student-teacher behaviour was used to provide support for, or against, the constructivist principles. The results show that the three VLEs meet the constructivist principles in very different ways and recommendations for design modifications are put forward

A Master Wittgensteinian Surveys Human Nature -A Review of Human Nature-the Categorial Framework by PMS Hacker (2010) (review revised 2019)

Materialism, reductionism, behaviorism, functionalism, dynamic systems theory and computationalism are popular views, but they were shown by Wittgenstein and more recently by Searle to be incoherent. The study of behavior encompasses all of human life but behavior is largely automatic and unconscious and even the conscious part, mostly expressed in language (which Wittgenstein equates with the mind), is not perspicuous, so it is critical to have a framework which Searle calls the Logical Structure of Rationality (LSR) and I call the Descriptive Psychology of Higher Order Thought (DPHOT). After summarizing the framework worked out by Wittgenstein and Searle, as extended by myself and by modern reasoning research, I comment on this first book in a trilogy on Human Nature by P.M.S. Hacker, the leading authority on Wittgenstein and one of the best modern philosophers. Those wishing a comprehensive up to date framework for human behavior from the modern two systems view may consult my book ‘The Logical Structure of Philosophy, Psychology, Mind and Language in Ludwig Wittgenstein and John Searle’ 2nd ed (2019). Those interested in more of my writings may see ‘Talking Monkeys--Philosophy, Psychology, Science, Religion and Politics on a Doomed Planet--Articles and Reviews 2006-2019 3rd ed (2019), The Logical Structure of Human Behavior (2019), and Suicidal Utopian Delusions in the 21st Century 4th ed (2019