A survey on knowledge-enhanced multimodal learning

Multimodal learning has been a field of increasing interest, aiming to combine various modalities in a single joint representation. Especially in the area of visiolinguistic (VL) learning multiple models and techniques have been developed, targeting a variety of tasks that involve images and text. VL models have reached unprecedented performances by extending the idea of Transformers, so that both modalities can learn from each other. Massive pre-training procedures enable VL models to acquire a certain level of real-world understanding, although many gaps can be identified: the limited comprehension of commonsense, factual, temporal and other everyday knowledge aspects questions the extendability of VL tasks. Knowledge graphs and other knowledge sources can fill those gaps by explicitly providing missing information, unlocking novel capabilities of VL models. In the same time, knowledge graphs enhance explainability, fairness and validity of decision making, issues of outermost importance for such complex implementations. The current survey aims to unify the fields of VL representation learning and knowledge graphs, and provides a taxonomy and analysis of knowledge-enhanced VL models

Visualising key information and communication technologies (ICT) indicators for children and young individuals in Europe

DGmap is an online interactive tool that visualises indicators drawn from large-scale European and international databases reflecting the use of information and communication technologies (ICT) amongst children and young individuals in Europe. A large number of indicators are estimated and visualised on an interactive map revealing convergences and divergences amongst European countries. Apart from its main feature, that of facilitating users to observe discrepancies between countries, the map offers the potentiality of downloading or customising country reports, information concerning the estimation of the indices and their values as spreadsheets, while covering a period from 2015 and onwards. DGmap also allows users to examine the evolution of each indicator through time for each country individually. Thus, the presented tool is a dynamic and constantly updated application that can serve as a major source of information for those interested in the use of digital technologies by children, adolescents, and young people in Europe

Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation

Visual Word Sense Disambiguation (VWSD) is a novel challenging task with the goal of retrieving an image among a set of candidates, which better represents the meaning of an ambiguous word within a given context. In this paper, we make a substantial step towards unveiling this interesting task by applying a varying set of approaches. Since VWSD is primarily a text-image retrieval task, we explore the latest transformer-based methods for multimodal retrieval. Additionally, we utilize Large Language Models (LLMs) as knowledge bases to enhance the given phrases and resolve ambiguity related to the target word. We also study VWSD as a unimodal problem by converting to text-to-text and image-to-image retrieval, as well as question-answering (QA), to fully explore the capabilities of relevant models. To tap into the implicit knowledge of LLMs, we experiment with Chain-of-Thought (CoT) prompting to guide explainable answer generation. On top of all, we train a learn to rank (LTR) model in order to combine our different modules, achieving competitive ranking results. Extensive experiments on VWSD demonstrate valuable insights to effectively drive future directions.Comment: Conference on Empirical Methods in Natural Language Processing (EMNLP) 202

Towards explainable evaluation of language models on the semantic similarity of visual concepts

Recent breakthroughs in NLP research, such as the advent of Transformer models have indisputably contributed to major advancements in several tasks. However, few works research robustness and explainability issues of their evaluation strategies. In this work, we examine the behavior of high-performing pre-trained language models, focusing on the task of semantic similarity for visual vocabularies. First, we address the need for explainable evaluation metrics, necessary for understanding the conceptual quality of retrieved instances. Our proposed metrics provide valuable insights in local and global level, showcasing the inabilities of widely used approaches. Secondly, adversarial interventions on salient query semantics expose vulnerabilities of opaque metrics and highlight patterns in learned linguistic representations

Βελτιστοποίηση Συναρτήσεων με Άμεσες Μεθόδους

Η διπλωματική εργασία έχει ως σημείο αναφοράς έναν σημαντικό κλάδο της Αριθμητικής Ανάλυσης που αφορά τη βελτιστοποίηση συναρτήσεων, με το ενδιαφέρον μας να επικεντρώνεται σε μεθόδους που δεν κάνουν χρήση τεχνικών του Διαφορικού Λογισμού. Η θεωρία βελτιστοποίησης και οι σχετικές τεχνικές καταλαμβάνουν μια μεγάλη περιοχή των εφαρμοσμένων μαθηματικών. Στην παρούσα μελέτη εξετάζουμε το θέμα τόσο από θεωρητικής, όσο και από υπολογιστικής άποψης, με υλοποίηση των σχετικών αλγορίθμων στο Matlab. Αρχικά δίνεται ο σχετικός συμβολισμός και προτείνεται ο κατάλληλος φορμαλισμός για το πρόβλημα που μας ενδιαφέρει. Στην συνέχεια αναλύεται το θεωρητικό υπόβαθρο για τις μεθόδους άμεσης αναζήτησης που εξετάζουμε. Ως διαφωτιστικό παράδειγμα, αναλύουμε με λεπτομέρεια τη σχετικά απλή μέθοδο Compass και δίνουμε τον κώδικα Matlab που την υλοποιεί. Στην συνέχεια θα εξετάσουμε τις άλλες τρεις μεθόδους (ενισχυμένη Compass, Box και Hooke – Jeeves) που θα χρησιμοποιήσουμε, αντιμετωπίζοντάς τις ως ειδικές περιπτώσεις της γενικής θεωρίας. Για κάθε μέθοδο, δίνουμε και πάλι τον κώδικα Matlab. Τέλος, εκτελούνται αριθμητικά πειράματα πάνω σε τέσσερις συναρτήσεις δοκιμής, προκειμένου να ελέγξουμε την αποτελεσματικότητα των μεθόδων, τόσο μεμονωμένα, όσο και συγκριτικά. Στη γενική του διατύπωση ένα πρόβλημα βελτιστοποίησης, είναι ένα πρόβλημα εύρεσης της καλύτερης δυνατής λύσης από ένα δεδομένο σύνολο διαθέσιμων επιλογών. Στην απλούστερη περίπτωση, συνίσταται στην ελαχιστοποίηση (ή μεγιστοποίηση) μιας πραγματικής συνάρτησης.The thesis has as a reference point an important branch of Numerical Analysis of the optimization function. Our interest focus on methods that do not use techniques of Differential Calculus. The optimization theory and related techniques occupy on a large area of ​​applied mathematics. In this study we examine the issue from both theoretical and computational point of view, with implementation of the relevant algorithms in Matlab. Originally the related symbolism is given and appropriate formalism is proposed for the problem of interest. The theoretical background is analyzed for direct search optimization methods. As an enlightening example, we analyze in detail the relatively simple method of Compass and give the Matlab code implementation. Then we will examine the other three calculation methods (enhanced Compass, Box and Hooke - Jeeves) that we will use, treating them as special cases of the general theory. For each method, we will provide the Matlab implementation code. Finally, numerical experiments performed on our four test functions in order to check the effectiveness of procedures, both individually and comparatively. In general, a formulation optimization problem is a problem of finding the best solution from a given set of available options. In the simplest case, our main research goal is to minimize (or maximize) a real function

Design, Development and Evaluation of 5G-Enabled Vehicular Services:The 5G-HEART Perspective

The ongoing transition towards 5G technology expedites the emergence of a variety of mobile applications that pertain to different vertical industries. Delivering on the key commitment of 5G, these diverse service streams, along with their distinct requirements, should be facilitated under the same unified network infrastructure. Consequently, in order to unleash the benefits brought by 5G technology, a holistic approach towards the requirement analysis and the design, development, and evaluation of multiple concurrent vertical services should be followed. In this paper, we focus on the Transport vertical industry, and we study four novel vehicular service categories, each one consisting of one or more related specific scenarios, within the framework of the “5G Health, Aquaculture and Transport (5G-HEART)” 5G PPP ICT-19 (Phase 3) project. In contrast to the majority of the literature, we provide a holistic overview of the overall life-cycle management required for the realization of the examined vehicular use cases. This comprises the definition and analysis of the network Key Performance Indicators (KPIs) resulting from high-level user requirements and their interpretation in terms of the underlying network infrastructure tasked with meeting their conflicting or converging needs. Our approach is complemented by the experimental investigation of the real unified 5G pilot’s characteristics that enable the delivery of the considered vehicular services and the initial trialling results that verify the effectiveness and feasibility of the presented theoretical analysis

A Balanced Translocation in Kallmann Syndrome Implicates a Long Noncoding RNA, RMST, as a GnRH Neuronal Regulator.

CONTEXT: Kallmann syndrome (KS) is a rare, genetically heterogeneous Mendelian disorder. Structural defects in KS patients have helped define the genetic architecture of gonadotropin-releasing hormone (GnRH) neuronal development in this condition. OBJECTIVE: Examine the functional role a novel structural defect affecting a long noncoding RNA (lncRNA), RMST, found in a KS patient. DESIGN: Whole genome sequencing, induced pluripotent stem cells and derived neural crest cells (NCC) from the KS patient were contrasted with controls. SETTING: The Harvard Reproductive Sciences Center, Massachusetts General Hospital Center for Genomic Medicine, and Singapore Genome Institute. PATIENT: A KS patient with a unique translocation, t(7;12)(q22;q24). INTERVENTIONS/MAIN OUTCOME MEASURE/RESULTS: A novel translocation was detected affecting the lncRNA, RMST, on chromosome 12 in the absence of any other KS mutations. Compared with controls, the patient's induced pluripotent stem cells and NCC provided functional information regarding RMST. Whereas RMST expression increased during NCC differentiation in controls, it was substantially reduced in the KS patient's NCC coincident with abrogated NCC morphological development and abnormal expression of several "downstream" genes essential for GnRH ontogeny (SOX2, PAX3, CHD7, TUBB3, and MKRN3). Additionally, an intronic single nucleotide polymorphism in RMST was significantly implicated in a genome-wide association study associated with age of menarche. CONCLUSIONS: A novel deletion in RMST implicates the loss of function of a lncRNA as a unique cause of KS and suggests it plays a critical role in the ontogeny of GnRH neurons and puberty

Analysis of ancestral and functionally relevant CD5 variants in systemic lupus erythematosus patients

OBJECTIVE: CD5 plays a crucial role in autoimmunity and is a well-established genetic risk factor of developing RA. Recently, evidence of positive selection has been provided for the CD5 Pro224-Val471 haplotype in East Asian populations. The aim of the present work was to further analyze the functional relevance of non-synonymous CD5 polymorphisms conforming the ancestral and the newly derived haplotypes (Pro224-Ala471 and Pro224-Val471, respectively) as well as to investigate the potential role of CD5 on the development of SLE and/or SLE nephritis. METHODS: The CD5 SNPs rs2241002 (C/T; Pro224Leu) and rs2229177 (C/T; Ala471Val) were genotyped using TaqMan allelic discrimination assays in a total of 1,324 controls and 681 SLE patients of Spanish origin. In vitro analysis of CD3-mediated T cell proliferative and cytokine response profiles of healthy volunteers homozygous for the above mentioned CD5 haplotypes were also analyzed. RESULTS: T-cell proliferation and cytokine release were significantly increased showing a bias towards to a Th2 profile after CD3 cross-linking of peripheral mononuclear cells from healthy individuals homozygous for the ancestral Pro224-Ala471 (CC) haplotype, compared to the more recently derived Pro224-Val471 (CT). The same allelic combination was statistically associated with Lupus nephritis. CONCLUSION: The ancestral Ala471 CD5 allele confers lymphocyte hyper-responsiveness to TCR/CD3 cross-linking and is associated with nephritis in SLE patients

POU6F2 mutation in humans with pubertal failure alters GnRH transcript expression

Idiopathic hypogonadotropic hypogonadism (IHH) is characterized by the absence of pubertal development and subsequent impaired fertility often due to gonadotropin-releasing hormone (GnRH) deficits. Exome sequencing of two independent cohorts of IHH patients identified 12 rare missense variants in POU6F2 in 15 patients. POU6F2 encodes two distinct isoforms. In the adult mouse, expression of both isoform1 and isoform2 was detected in the brain, pituitary, and gonads. However, only isoform1 was detected in mouse primary GnRH cells and three immortalized GnRH cell lines, two mouse and one human. To date, the function of isoform2 has been verified as a transcription factor, while the function of isoform1 has been unknown. In the present report, bioinformatics and cell assays on a human-derived GnRH cell line reveal a novel function for isoform1, demonstrating it can act as a transcriptional regulator, decreasing GNRH1 expression. In addition, the impact of the two most prevalent POU6F2 variants, identified in five IHH patients, that were located at/or close to the DNA-binding domain was examined. Notably, one of these mutations prevented the repression of GnRH transcripts by isoform1. Normally, GnRH transcription increases as GnRH cells mature as they near migrate into the brain. Augmentation earlier during development can disrupt normal GnRH cell migration, consistent with some POU6F2 variants contributing to the IHH pathogenesis