Inferring implicit relevance from physiological signals

Ongoing growth in data availability and consumption has meant users are increasingly faced with the challenge of distilling relevant information from an abundance of noise. Overcoming this information overload can be particularly difficult in situations such as intelligence analysis, which involves subjectivity, ambiguity, or risky social implications. Highly automated solutions are often inadequate, therefore new methods are needed for augmenting existing analysis techniques to support user decision making. This project investigated the potential for deep learning to infer the occurrence of implicit relevance assessments from users' biometrics. Internal cognitive processes manifest involuntarily within physiological signals, and are often accompanied by 'gut feelings' of intuition. Quantifying unconscious mental processes during relevance appraisal may be a useful tool during decision making by offering an element of objectivity to an inherently subjective situation. Advances in wearable or non-contact sensors have made recording these signals more accessible, whilst advances in artificial intelligence and deep learning have enhanced the discovery of latent patterns within complex data. Together, these techniques might make it possible to transform tacit knowledge into codified knowledge which can be shared. A series of user studies recorded eye gaze movements, pupillary responses, electrodermal activity, heart rate variability, and skin temperature data from participants as they completed a binary relevance assessment task. Participants were asked to explicitly identify which of 40 short-text documents were relevant to an assigned topic. Investigations found this physiological data to contain detectable cues corresponding with relevance judgements. Random forests and artificial neural networks trained on features derived from the signals were able to produce inferences with moderate correlations with the participants' explicit relevance decisions. Several deep learning algorithms trained on the entire physiological time series data were generally unable to surpass the performance of feature-based methods, and instead produced inferences with low correlations with participants' explicit personal truths. Overall, pupillary responses, eye gaze movements, and electrodermal activity offered the most discriminative power, with additional physiological data providing diminishing or adverse returns. Finally, a conceptual design for a decision support system is used to discuss social implications and practicalities of quantifying implicit relevance using deep learning techniques. Potential benefits included assisting with introspection and collaborative assessment, however quantifying intrinsically unknowable concepts using personal data and abstruse artificial intelligence techniques were argued to pose incommensurate risks and challenges. Deep learning techniques therefore have the potential for inferring implicit relevance in information-rich environments, but are not yet fit for purpose. Several avenues worthy of further research are outlined