Affective Image Content Analysis: Two Decades Review and New Perspectives

Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.Comment: Accepted by IEEE TPAM

NEmo – news that triggers emotions, an affectively-annotated dataset of gun violence news

Given our society’s increased exposure to multimedia formats on social media platforms, efforts to understand how digital content impacts people’s emotions are burgeoning. As such, we introduce a U.S. gun violence news dataset that contains news headline and image pairings from 840 news articles with 15K high-quality, crowdsourced annotations on emotional responses to the news pairings. We created three experimental conditions for the annotation process: two with a single modality (headline or image only), and one multimodal (headline and image together). In contrast to prior works on affectively-annotated data, our dataset includes annotations on the dominant emotion experienced with the content, the intensity of the selected emotion and an open-ended, written component. By collecting annotations on different modalities of the same news content pairings, we explore the relationship between image and text influence on human emotional response. We offer initial analysis on our dataset, showing the nuanced affective differences that appear due to modality and individual factors such as political leaning and media consumption habits. Our dataset is made publicly available to facilitate future research in affective computing.http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.267.pdfPublished versio

Backwards is the way forward: feedback in the cortical hierarchy predicts the expected future

Clark offers a powerful description of the brain as a prediction machine, which offers progress on two distinct levels. First, on an abstract conceptual level, it provides a unifying framework for perception, action, and cognition (including subdivisions such as attention, expectation, and imagination). Second, hierarchical prediction offers progress on a concrete descriptive level for testing and constraining conceptual elements and mechanisms of predictive coding models (estimation of predictions, prediction errors, and internal models)

Naturalistic language comprehension : a fMRI study on semantics in a narrative context

Semantiikka tutkii kieleen sisältyviä merkityksiä, joita tarvitaan kielen ymmärryksessä. Kuinka aivomme käsittelevät semantiikkaa ja kuinka ymmärrämme erityisesti luonnollisessa muodossa olevaa kieltä, on vielä aivotutkijoille epäselvää. Tässä tutkimuksessa kysyttiin, miten laajemmassa kontekstissa, narratiivissa, olevan kielen ymmärrys ja semanttinen prosessointi heijastuu aivojen aktiivisuuteen. Koehenkilöt kuulivat narratiivin toiminnallisen magneettiresonanssikuvantamisen (fMRI) aikana. Narratiivin semanttinen sisältö mallinnettiin laskennallisesti word2vec algoritmin avulla, ja tätä mallia verrattiin veren happitasosta riippuvaiseen (BOLD) aivosignaaliin ridge regression avulla vokseli kerrallaan. Lähestymistavalla saatiin eristettyä yksityiskohtaisempaa tietoa jatkuvan stimuluksen aivodatasta perustuen kielen semanttiseen sisältöön. Subjektien välinen BOLD-signaalin korrelaatio (ISC) itsessään paljasti molempien aivopuoliskojen osallistuvan kielen ymmärrykseen laajasti. Alueellista päällekkäisyyttä löytyi muiden aivoverkostojen kanssa, jotka vastaavat mm. mentalisaatiosta, muistista ja keskittymiskyvystä, mikä viittaa kielen ymmärryksen vaativan myös muiden kognition osien toimintaa. Ridge regression tulokset viittaavat bilateraalisten pikkuaivojen, superiorisen, keskimmäisen sekä mediaalisen etuaivokuoren poimujen, inferiorisen ja mediaalisen parietaalikuoren sekä visuaalikuoren, sekä oikean temporaalikuoren osallistuvan narratiivin semanttiseen prosessointiin aivoissa. Aiempi semantiikan tutkimus on tuottanut samankaltaisia tuloksia, joten word2vec vaikuttaisi tämän tutkimuksen perusteella mallintavan semantiikkaa riittävän hyvin aivotutkimuksen tarpeisiin. Tutkimuksen perusteella molemmat aivopuoliskot osallistuvat kielen laajemman kontekstin käsittelyyn, ja semantiikka nähdään aktivaationa eri puolilla aivokuorta. Nämä aktiivisuudet ovat mahdollisesti riippuvaisia kielen sisällöstä, mutta miten paljon kielen sisältö vaikuttaa eri aivoalueiden osallistumiseen kielen semanttisessa prosessoinnissa, on vielä avoin tutkimuskysymys.Semantics is a study of meaning in language and basis for language comprehension. How these phenomena are processed in the brain is still unclear especially in naturalistic context. In this study, naturalistic language comprehension, and how semantic processing in a narrative context is reflected in brain activity were investigated. Subjects were measured with functional magnetic resonance imaging (fMRI) while listening to a narrative. The semantic content of the narrative was modelled computationally with word2vec and compared to voxel-wise blood-oxygen-level dependent (BOLD) brain signal time courses using ridge regression. This approach provides a novel way to extract more detailed information from the brain data based on semantic content of the stimulus. Inter-subject correlation (ISC) of voxel-wise BOLD signals alone showed both hemispheres taking part in language comprehension. Areas involved in this task overlapped with networks of mentalisation, memory and attention suggesting comprehension requiring other modalities of cognition for its function. Ridge regression suggested cerebellum, superior, middle and medial frontal, inferior and medial parietal and visual cortices bilaterally and temporal cortex on right hemisphere having a role in semantic processing of the narrative. As similar results have been found in previous research on semantics, word2vec appears to model semantics sufficiently and is an applicable tool in brain research. This study suggests contextual language recruiting brain areas in both hemispheres and semantic processing showing as distributed activity on the cortex. This activity is likely dependent on the content of language, but further studies are required to distinguish how strongly brain activity is affected by different semantic contents

Advances in Clinical Neurophysiology

Including some of the newest advances in the field of neurophysiology, this book can be considered as one of the treasures that interested scientists would like to collect. It discusses many disciplines of clinical neurophysiology that are, currently, crucial in the practice as they explain methods and findings of techniques that help to improve diagnosis and to ensure better treatment. While trying to rely on evidence-based facts, this book presents some new ideas to be applied and tested in the clinical practice. Advances in Clinical Neurophysiology is important not only for the neurophysiologists but also for clinicians interested or working in wide range of specialties such as neurology, neurosurgery, intensive care units, pediatrics and so on. Generally, this book is written and designed to all those involved in, interpreting or requesting neurophysiologic tests

On Improving Generalization of CNN-Based Image Classification with Delineation Maps Using the CORF Push-Pull Inhibition Operator

Deployed image classification pipelines are typically dependent on the images captured in real-world environments. This means that images might be affected by different sources of perturbations (e.g. sensor noise in low-light environments). The main challenge arises by the fact that image quality directly impacts the reliability and consistency of classification tasks. This challenge has, hence, attracted wide interest within the computer vision communities. We propose a transformation step that attempts to enhance the generalization ability of CNN models in the presence of unseen noise in the test set. Concretely, the delineation maps of given images are determined using the CORF push-pull inhibition operator. Such an operation transforms an input image into a space that is more robust to noise before being processed by a CNN. We evaluated our approach on the Fashion MNIST data set with an AlexNet model. It turned out that the proposed CORF-augmented pipeline achieved comparable results on noise-free images to those of a conventional AlexNet classification model without CORF delineation maps, but it consistently achieved significantly superior performance on test images perturbed with different levels of Gaussian and uniform noise