New approaches to interactive multimedia content retrieval from different sources

Mención Internacional en el título de doctorInteractive Multimodal Information Retrieval systems (IMIR) increase the capabilities of traditional search systems with the ability to retrieve information in different types (modes) and from different sources. The increase in online content while diversifying means of access to information (phones, tablets, smart watches) encourages the growing need for this type of system. In this thesis a formal model for describing interactive multimodal information retrieval systems querying various information retrieval engines has been defined. This model includes formal and widespread definition of each component of an IMIR system, namely: multimodal information organized in collections, multimodal query, different retrieval engines, a source management system (handler), a results management module (fusion) and user interactions. This model has been validated in two stages. The first, in a use case focused on information retrieval on sports. A prototype that implements a subset of the features of the model has been developed: a multimodal collection that is semantically related, three types of multimodal queries (text, audio and text + image), six different retrieval engines (question answering, full-text search, search based on ontologies, OCR in image, object detection in image and audio transcription), a strategy for source selection based on rules defined by experts, a strategy of combining results and recording of user interactions. NDCG (normalized discounted cumulative gain) has been used for comparing the results obtained for each retrieval engine. These results are: 10,1% (Question answering), 80% (full text search) and 26;8% (ontology search). These results are on the order of works of the state of art considering forums like CLEF. When the retrieval engine combination is used, the information retrieval performance increases by a percentage gain of 771,4% with question answering, 7,2% with full text search and 145,5% with Ontology search. The second scenario is focused on a prototype retrieving information from social media in the health domain. A prototype has been developed which is based on the proposed model and integrates health domain social media user-generated information, knowledge bases, query, retrieval engines, sources selection module, results' combination module and GUI. In addition, the documents included in the retrieval system have been previously processed by a process that extracts semantic information in health domain. In addition, several adaptation techniques applied to the retrieval functionality of an IMIR system have been defined by analyzing past interactions using decision trees, neural networks and clusters. After modifying the sources selection strategy (handler), the system has been reevaluated using classification techniques. The same queries and relevance judgments done by users in the sports domain prototype will be used for this evaluation. This evaluation compares the normalized discounted cumulative gain (NDCG) measure obtained with two different approaches: the multimodal system using predefined rules and the same multimodal system once the functionality is adapted by past user interactions. The NDCG has shown an improvement between -2,92% and 2,81% depending on the approaches used. We have considered three features to classify the approaches: (i) the classification algorithm; (ii) the query features; and (iii) the scores for computing the orders of retrieval engines. The best result is obtained using probabilities-based classification algorithm, the retrieval engines ranking generated with Averaged-Position score and the mode, type, length and entities of the query. Its NDCG value is 81,54%.Los Sistemas Interactivos de Recuperación de Información Multimodal (IMIR) incrementan las capacidades de los sistemas tradicionales de búsqueda con la posibilidad de recuperar información de diferentes tipos (modos) y a partir de diferentes fuentes. El incremento del contenido en internet a la vez que la diversificación de los medios de acceso a la información (móviles, tabletas, relojes inteligentes) fomenta la necesidad cada vez mayor de este tipo de sistemas. En esta tesis se ha definido un modelo formal para la descripción de sistemas de recuperación de información multimodal e interactivos que consultan varios motores de recuperación. Este modelo incluye la definición formal y generalizada de cada componente de un sistema IMIR, a saber: información multimodal organizada en colecciones, consulta multimodal, diferentes motores de recuperación, sistema de gestión de fuentes (handler), módulo de gestión de resultados (fusión) y las interacciones de los usuarios. Este modelo se ha validado en dos escenarios. El primero, en un caso de uso focalizado en recuperación de información relativa a deportes. Se ha desarrollado un prototipo que implementa un subconjunto de todas las características del modelo: una colección multimodal que se relaciona semánticamente, tres tipos de consultas multimodal (texto, audio y texto + imagen), seis motores diferentes de recuperación (búsqueda de respuestas, búsqueda de texto completo, búsqueda basada en ontologías, OCR en imagen, detección de objetos en imagen y transcripción de audio), una estrategia de selección de fuentes basada en reglas definidas por expertos, una estrategia de combinación de resultados y el registro de las interacciones. Se utiliza la medida NDCG (normalized discounted cumulative gain) para describir los resultados obtenidos por cada motor de recuperación. Estos resultados son: 10,1% (Question Answering), 80% (Búsqueda a texto completo) y 26,8% (Búsqueda en ontologías). Estos resultados están en el orden de los trabajos del estado de arte considerando foros como CLEF (Cross-Language Evaluation Forum). Cuando se utiliza la combinación de motores de recuperación, el rendimiento de recuperación de información se incrementa en un porcentaje de ganancia de 771,4% con Question Answering, 7,2% con Búsqueda a texto completo y 145,5% con Búsqueda en ontologías. El segundo escenario es un prototipo centrado en recuperación de información de medios sociales en el dominio de salud. Se ha desarrollado un prototipo basado en el modelo propuesto y que integra información del dominio de salud generada por el usuario en medios sociales, bases de conocimiento, consulta, motores de recuperación, módulo de selección de fuentes, módulo de combinación de resultados y la interfaz gráfica de usuario. Además, los documentos incluidos en el sistema de recuperación han sido previamente anotados mediante un proceso de extracción de información semántica del dominio de salud. Además, se han definido técnicas de adaptación de la funcionalidad de recuperación de un sistema IMIR analizando interacciones pasadas mediante árboles de decisión, redes neuronales y agrupaciones. Una vez modificada la estrategia de selección de fuentes (handler), se ha evaluado de nuevo el sistema usando técnicas de clasificación. Las mismas consultas y juicios de relevancia realizadas por los usuarios en el primer prototipo sobre deportes se han utilizado para esta evaluación. La evaluación compara la medida NDCG (normalized discounted cumulative gain) obtenida con dos enfoques diferentes: el sistema multimodal usando reglas predefinidas y el mismo sistema multimodal una vez que la funcionalidad se ha adaptado por las interacciones de usuario. El NDCG ha mostrado una mejoría entre -2,92% y 2,81% en función de los métodos utilizados. Hemos considerado tres características para clasificar los enfoques: (i) el algoritmo de clasificación; (ii) las características de la consulta; y (iii) las puntuaciones para el cálculo del orden de los motores de recuperación. El mejor resultado se obtiene utilizando el algoritmo de clasificación basado en probabilidades, las puntuaciones para los motores de recuperación basados en la media de la posición del primer resultado relevante y el modo, el tipo, la longitud y las entidades de la consulta. Su valor de NDCG es 81,54%.Programa Oficial de Doctorado en Ciencia y Tecnología InformáticaPresidente: Ana García Serrano.- Secretario: María Belén Ruiz Mezcua.- Vocal: Davide Buscald

Frequency effects in the processing of verbs and argument structure: Evidence from adults with and without acquired aphasia

Introduction In usage-based approaches to language, grammar is viewed as an emergent phenomenon that derives from humans’ repeated exposure to individual instances of particular linguistic expressions (Bybee, 2006). Goldberg’s (1995) construction grammar is a version of usage-based grammar that treats language as an inventory of form-meaning pairings, termed constructions. Usage-based approaches to language predict that factors of language use, such as frequency of occurrence, affect processing at every level of the linguistic system, from sounds to sentences. This approach is gaining increasing recognition in the field of aphasiology, where sentence-level frequency effects have historically been described in terms of deficits (Gahl & Menn, 2016). The current research adopts a usage-based approach to language and contributes new data on the topic of verb and sentence processing in typical adults and adults with acquired aphasia. Aims This research investigated the effects of two frequency-based properties of verbs on language processing in adults, including the frequency of a verb as a single word, termed lexical frequency, and the frequency of a verb in a particular syntactic construction, termed construction frequency. Specifically, this project aimed: (1) to examine the effect of construction frequency and lexical frequency on sentence processing in adults; (2) to explore whether the pattern of performance from adults with acquired aphasia was similar to or divergent from the performance of typical adults; and (3) to consider how residual linguistic capabilities in participants with aphasia affected their performance in experimental tasks. Methods In Phase 1, 20 typical adults and four adults with acquired aphasia took part in a verbal fluency task in which they named verbs that could occur in eight unique syntactic constructions. Noun phrases were encoded as pronouns, so no semantic activation was available from the lexemes contained in sentence stimuli, and a blank space stood in place of the verb. For example, a sentence corresponding to the conative construction was presented as you ___ at us. In Phase 2, 90 typical adults and 14 adults with acquired aphasia took part in a grammaticality judgement task and a sentence completion task. Participants silently read sentences like those in Phase 1 and were subsequently presented with a written verb. In the grammaticality judgement task, participants decided whether or not the verb could occur in the sentence stimulus. In the sentence completion task, participants replaced the blank space in the sentence stimulus with the given verb and produced the entire sentence aloud. Participants’ number of target responses and response times were measured in each task. The frequency of verbs in Phase 2 varied along two dimensions. These independent variables included construction frequency and lexical frequency, each of which had two levels, namely, high frequency and low frequency. These four groups resulted in a factorial design, where conditions differed with respect to levels of construction frequency and lexical frequency. Results In Phase 1, the number of times typical participants generated verbs in response to syntactic constructions was more strongly related to verbs’ construction frequency than lexical frequency, for most constructions. Sentence stimuli successfully elicited verbs from participants with aphasia. In Phase 2, typical participants showed an effect of construction frequency in the grammaticality judgement task and an effect of lexical frequency in the sentence completion task. These effects were moderated by construction and interactions. In general, group-level results from participants with aphasia were consistent with findings from typical participants. Some individuals with aphasia showed frequency effects to a greater or lesser extent than typical participants. Conclusion Results suggest that at the sentence level, the frequency of verbs as single words and the frequency of verbs in particular syntactic contexts affects language processing, depending on task demands. Findings confirm the predicted effect of linguistic experience on language use. Importantly, this project extends the number of investigations of pathological language undertaken in a usage-based linguistic framework. Results from participants with aphasia are discussed with reference to treatments for sentence processing deficits in aphasia, item selection for those treatments and theories of agrammatism