Framing digital image credibility: image manipulation problems, perceptions and solutions

Image manipulation is subverting the credibility of photographs as a whole. Currently there is no practical solution for asserting the authenticity of a photograph. People express their concern about this when asked but continue to operate in a ‘business as usual’ fashion. While a range of digital forensic technologies has been developed to address falsification of digital photographs, such technologies begin with ‘sourceless’ images and conclude with results in equivocal terms of probability, while not addressing the meaning and content contained within the image. It is interesting that there is extensive research into computer-based image forgery detection, but very little research into how we as humans perceive, or fail to perceive, these forgeries when we view them. The survey, eye-gaze tracking experiments and neural network analysis undertaken in this research contribute to this limited pool of knowledge. The research described in this thesis investigates human perceptions of images that are manipulated and, by comparison, images that are not manipulated. The data collected, and their analyses, demonstrate that humans are poor at identifying that an image has been manipulated. I consider some of the implications of digital image manipulation, explore current approaches to image credibility, and present a potential digital image authentication framework that uses technology and tools that exploit social factors such as reputation and trust to create a framework for technologically packaging/wrapping images with social assertions of authenticity, and surfaced metadata information. The thesis is organised into 6 chapters. Chapter 1: Introduction I briefly introduce the history of photography, highlighting its importance as reportage, and discuss how it has changed from its introduction in the early 19th century to today. I discuss photo manipulation and consider how it has changed along with photography. I describe the relevant literature on the subject of image authentication and the use of eye gaze tracking and neural nets in identifying the role of human vision in image manipulation detection, and I describe my area of research within this context. Chapter 2: Literature review I describe the various types of image manipulation, giving examples, and then canvas the literature to describe the landscape of image manipulation problems and extant solutions, namely: • the nature of image manipulation, • investigations of human perceptions of image manipulation, • eye gaze tracking and manipulated images, • known efforts to create solutions to the problem of preserving unadulterated photographic representations and the meanings they hold. Finally, I position my research activities within the context of the literature. Chapter 3: The research I describe the survey and experiments I undertook to investigate attitudes toward image manipulation, research human perceptions of manipulated and unmanipulated images, and to trial elements of a new wrapper-style file format that I call .msci (mobile self-contained image), designed to address image authenticity issues. Methods, results and discussion for each element are presented in both explanatory text and by presentation of papers resulting from the experiments. Chapter 4: Analysis of eye gaze data using classification neural networks I describe pattern classifying neural network analysis applied to selected data obtained from the experiments and the insights this analysis provided into the opaque realm of cognitive perception as seen through the lens of eye gaze. Chapter 5: Discussion I synthesise and discuss the outcomes of the survey and experiments. I discuss the outcomes of this research, and consider the need for a distinction between photographs and photo art. I offer a theoretical formula within which the overall authenticity of an image can be assessed. In addition I present a potential image authentication framework built around the .msci file format, designed in consideration of my investigation of the requirements of the image manipulation problem space and the experimental work undertaken in this research. Chapter 6: Conclusions and future work This thesis concludes with a summary of the outcomes of my research, and I consider the need for future experimentation to expand on the insights gained to date. I also note some ways forward to develop an image authentication framework to address the ongoing problem of image authenticity

Digital Manipulation of Human Faces: Effects on Emotional Perception and Brain Activity

The study of human face-processing has granted insight into key adaptions across various social and biological functions. However, there is an overall lack of consistency regarding digital alteration styles of human-face stimuli. In order to investigate this, two independent studies were conducted examining unique effects of image construction and presentation. In the first study, three primary forms of stimuli presentation styles (color, black and white, cutout) were used across iterations of non-thatcherized/thatcherized and non-inverted/inverted presentations. Outcome measures included subjective reactions measured via ratings of perceived “grotesqueness,” and objective outcomes of N170 event-related potentials (ERPs) measured via encephalography. Results of subjective measures indicated that thatcherized images were associated with an increased level of grotesque perception, regardless of overall condition variant and inversion status. A significantly larger N170 component was found in response to cutout-style images of human faces, thatcherized images, and inverted images. Results suggest that cutout image morphology may be considered a well-suited image presentation style when examining ERPs and facial processing of otherwise unaltered human faces. Moreover, less emphasis can be placed on decision making regarding main condition morphology of human face stimuli as it relates to negatively valent reactions. The second study explored commonalities between thatcherized and uncanny images. The purpose of the study was to explore commonalities between these two styles of digital manipulation and establish a link between previously disparate areas of human-face processing research. Subjective reactions to stimuli were measured via participant ratings of “off-putting.” ERP data were gathered in order to explore if any unique effects emerged via N170 and N400 presentations. Two main “morph continuums” of stimuli, provided by Eduard Zell (see Zell et al., 2015), with uncanny features were utilized. A novel approach of thatcherizing images along these continuums was used. thatcherized images across both continuums were regarded as more off-putting than non-thatcherized images, indicating a robust subjective effect of thatcherization that was relatively unimpacted by additional manipulation of key featural components. Conversely, results from brain activity indicated no significant differences of N170 between level of shape stylization and their thatcherized counterparts. Unique effects between continuums and exploratory N400 results are discussed

"Thinstagram" : image content and observer body satisfaction influence the when and where of eye movements during Instagram image viewing

Selective Exposure Theory (Aruguete & Calvo, 2018; Bigné et al., 2020) suggests that on social media, viewers pay most attention to content which aligns with their values and preferences. Individuals engage in self-assessment by comparing themselves to others (Social comparison theory: Festinger, 1954) . We predicted that the characteristics of Instagram arrays and participants' own body satisfaction would combine to influence their visual processing of computer-based images. A 3 (Body Shape: Underweight, Average, Overweight) × 2 (Body Part: Face-only; Body-only) repeated measures design was used. We recruited 60 (young) women to view arrays of images as displayed on Instagram [M age = 20.75 years, SD age = 2.74 years]. A separate, naïve group of 37 participants rated 165 stimulus images on a scale of under-to-over-weight. These normed images were used to create artificial, ecologically-valid 3 × 4 Instagram image arrays containing two of each type of stimulus image. We recorded participants' eye movements with a high degree of spatial and temporal resolution while participants freely engaged with these arrays. We then collected participants' body satisfaction data (Slade et al., 1990). Results demonstrated inter-relationships between eye movement behaviour and Body Shape, Body Part, and body satisfaction. In short, both bottom-up stimulus characteristics and top-down satisfaction impacted measures of processing. Image content was particularly relevant to ‘when’ measures of processing time, whereas body satisfaction was more-influential upon ‘where’ measurements (fixations counts, number of visits per stimulus image). Our study is the first of its kind to show such effects. Future research is needed to understand such effects in clinical and/or non-female users of Instagram and other platforms

Eye Tracking to Support eLearning

Online eLearning environments to support student learning are of growing importance. Students are increasingly turning to online resources for education; sometimes in place of face-to-face tuition. Online eLearning extends teaching and learning from the classroom to a wider audience with different needs, backgrounds, and motivations. The one-size-fits-all approach predominately used is not effective for catering to the needs of all students. An area of the increasing diversity is the linguistic background of readers. More students are reading in their non-native language. It has previously been established that first English language (L1) students read differently to second English language (L2) students. One way of analysing this difference is by tracking the eyes of readers, which is an effective way of investigating the reading process. In this thesis we investigate the question of whether eye tracking can be used to make learning via reading more effective in eLearning environments. This question is approached from two directions; first by investigating how eye tracking can be used to adapt to individual student’s understanding and perceptions of text. The second approach is analysing a cohort’s reading behaviour to provide information to the author of the text and any related comprehension questions regarding their suitability and difficulty. To investigate these questions, two user studies were carried out to collect eye gaze data from both L1 and L2 readers. The first user study focussed on how different presentation methods of text and related questions affected not only comprehension performance but also reading behaviour and student perceptions of performance. The data from this study was used to make predictions of reading comprehension that can be used to make eLearning environments adaptive, in addition to providing implicit feedback about the difficulty of text and questions. In the second study we investigate the effects of text readability and conceptual difficulty on eye gaze, prediction of reading comprehension, and perceptions. This study showed that readability affected the eye gaze of L1 readers and conceptual difficulty affected the eye gaze of L2 readers. The prediction accuracy of comprehension was consequently increased for the L1 group by increased difficulty in readability, whereas increased difficulty in conceptual level corresponded to increased accuracy for the L2 group. Analysis of participants’ perceptions of complexity revealed that readability and conceptual difficulty interact making the two variables hard for the reader to disentangle. Further analysis of participants’ eye gaze revealed that both the predefined and perceived text complexity affected eye gaze. We therefore propose using eye gaze measures to provide feedback about the implicit reading difficulty of texts read. The results from both studies indicate that there is enormous potential in using eye tracking to make learning via reading more effective in eLearning environments. We conclude with a discussion of how these findings can be applied to improve reading within eLearning environments. We propose an adaptive eLearning architecture that dynamically presents text to students and provides information to authors to improve the quality of texts and questions

Meanness and Affective Processing: A Meta-Analysis of EEG Findings on Emotional Face Processing in Individuals with Psychopathic Traits

The triarchic model (Patrick, Fowles, & Krueger, 2009) conceptualizes psychopathy as a multidimensional construct encompassing three biobehavioral dimensions: meanness, boldness, and disinhibition. The biological correlates of meanness, which encompasses low empathy, shallow affect, and lack of guilt or remorse, are currently less well elucidated than boldness or disinhibition (Patrick & Drislane, 2015). At the behavioral level, meanness is related to decreased accuracy on tasks involving facial and emotion recognition (Brislin et al., 2018). Emotional face processing can be examined on a neurophysiological level using event-related potentials (ERPs) such as N170, P200, and LPP (Shannon et al., 2013). Research indicates the magnitude of these responses may be modulated by psychopathic meanness (Clark et al., 2019); however, discrepant findings have also been reported. Therefore, the current study performed random-effects model meta-analyses of nine studies meeting study inclusion criteria to provide an overall effect size for the association between meanness and affective face processing ERPs across studies. Results of the meta-analysis indicated a significant effect for N170 amplitude and meanness when processing fearful faces (r = 0.18). Significant effects were not found for N170 amplitude when processing angry or happy faces, nor were significant effects found for LPP and P200 amplitudes when processing fearful faces. Meta-regression analyses indicated the type of facial stimuli utilized across studies was significant in explaining some between-study heterogeneity of the N170-fear meta-analysis model. Through examining physiological indicators of meanness, the current study contributes to ongoing research on the etiology of psychopathy and may guide future research in establishing a multi-domain framework for the measurement of psychopathy

The relationship between attentional bias and body image disturbance

Body image disturbance is a both a risk factor for, and a symptom of, many eating disorders and refers to the misperception of and dissatisfaction with one’s own body. Some studies show that women with high body dissatisfaction direct more attention to low weight bodies. This attentional bias can result in the overestimation of body size via body size adaptation—a temporary perceptual shift experienced after exposure to extreme stimuli. Therefore, attention may have a causal role in body image disturbance. In this thesis, I test the effects of attentional bias to bodies of different sizes on body size adaptation and body dissatisfaction using a training dot probe task (Chapter 2) and a training visual search task (Chapter 5). I test the association between body dissatisfaction and attentional bias to low weight bodies using an assessment version of the dot probe task (Chapter 3), a systematic review and meta-analysis (Chapter 4) that synthesises the results of Chapters 2 and 3 with 30 additional eligible studies, and an assessment version of the Attentional Response to Distal vs. Proximal Emotional Information (ARDPEI) task (Chapter 6). From this research, I conclude that gaze tracking studies do provide evidence for a positive association between body dissatisfaction and attentional bias to low weight bodies in women. Women with high (compared to low) body dissatisfaction direct more gaze towards low weight female body stimuli. However, reaction time measures do not provide evidence for this association and instead demonstrate poor reliability as measures of individual differences in attention. This thesis does not provide strong evidence for an effect of attentional bias to bodies of different sizes on body size adaptation or body dissatisfaction; however, given attention was measured using reaction times, future research using gaze tracking should be conducted to further explore the effect of attentional bias to bodies of different sizes on body size adaptation or body dissatisfaction

Social perception of facial cues of adiposity

Previous research suggests that facial characteristics associated with body mass index (BMI) play an important role in health and attractiveness judgments of faces. However, very little work has investigated the factors that predict individual differences in preferences for facial cues of adiposity or how these individual differences are related to social outcomes. In light of the above, the first two empirical chapters of this thesis investigated the relationships between individual differences in preferences for facial cues of adiposity and (1) the BMI of men’s and women’s actual romantic partners and (2) disgust sensitivity. Analyses suggested that people with particularly strong preferences for slim-looking faces were more likely to have partners with low BMI and that men, but not women, who scored higher on pathogen disgust showed stronger aversions to faces displaying cues associated with high BMI. The third chapter investigated how people integrate information from shape cues of adiposity and information from skin color when judging the health and attractiveness of faces. Analyses showed that preferences for cues of low BMI were particularly strong when assessing faces displaying skin color cues associated with the absence of illness. These results suggest that integrating information from shape cues of adiposity and information from skin color could allow people to distinguish between individuals with low BMI because they are healthy and those with low BMI due to illness. Most research investigating the role of facial cues of adiposity in social perception has focused on the possible role of facial adiposity as a health cue. However, it is also possible that facial cues of adiposity contain other types of information, such as information about a person’s reported sociosexual orientation (openness to short-term, uncommitted sexual relationships). To explore this issue, the fourth empirical chapter of my thesis investigated the relationship between facial correlates of BMI and women’s sociosexual orientation. Although analyses suggested that slimmer women reported greater openness to short-term, uncommitted sexual relationships, the observed relationships were weak and, thus, unlikely to play an important role in social interactions. Together these studies support the claim that responses to facial cues of adiposity are related to romantic partner choice and function to distinguish between healthy and unhealthy individuals

Finding the Hidden: Detecting Atypical Affective States from Physiological Signals

In cognitive science, intuition is described as a strategy of processing information that relies on people's instinctive and emotional criteria. When compared with the deliberate choices made after conscious reasoning, the quick and intuitive decision making strategies can be more effective. The intuitive thinking provokes changes in human physiological responses which can be measured by sensors. Utilising physiological reactions, previous work shows that atypical patterns such as emotion expressions and image manipulations can be identified. This thesis expands the exploration to examine whether more atypical human behaviour can be recognised from physiological signals. The examined subtly atypical behaviour includes depression, doubt and deception, Depression is a serious chronic mental disease and is considered as an atypical health condition in people. Doubt is defined as a non-deliberate attempt to mislead others and is a passive form of deception, representing an atypicality from honest behaviours. Deception is a more purposeful attempt to deceive, and thus is a distinct type of atypicality than honest communication. Through examining physiological reactions from presenters who have a particular atypical behaviour or condition, and observers who view behaviours of presenters, this research aims to recognise atypicality in human behaviour. A collection of six user studies are conducted. In two user studies, presenters are asked to conduct doubting and deceiving behaviours, while the remaining user studies involve observers watching behaviours of presenters who suffer from depression, have doubt, or have conducted deception. Physiological reactions of both presenters and observers are collected, including Blood Volume Pulse, Electrodermal Activity, Skin Temperature and Pupillary Responses. Observers are also asked to explicitly evaluate whether the viewed presenters were being depressed, doubting, or deceiving. Investigations upon physiological data in this thesis finds that detectable cues corresponding with depression, doubt and deception can be found. Viewing depression provokes visceral physiological reactions in observers that can be measured. Such physiological responses can be used to derive features for machine learning models to accurately distinguish between healthy individuals and people with depression. By contrast, depression does not provoke strong conscious recognition in observers, resulting in a conscious evaluation accuracy slightly above chance level. Similar results are also found in detecting doubt and deception. People with doubt and deceit elicit consistent physiological reactions within themselves. These bodily responses can be utilised by machine learning models or deep learning models to recognise doubt or deception. The doubt and deceit in presenters can also be recognised using physiological signals in observers, with excellent recognition rates which are higher when compared with the conscious judgments from the same group of observers. The results indicate that atypicality in presenters can both be captured by physiological signals of presenters and observers. Presenters' physiological reactions contribute to higher recognition of atypicality, but observers' physiological responses can serve as a comparable alternative. The awareness of atypicality among observers happens physiologically, so can be used by machine learning models, even when they do not reach the consciousness of the person. The research findings lead to a further discussion around the implications of observers' physiological responses. Decision support applications which utilise a quantifiable measure of people's unconscious and intuitive 'gut feeling' can be developed based on the work reported here to assist people with medical diagnosis, information credibility evaluation, and criminal detection. Further research suggests exploring more atypical behaviours in the wild