Deep Architectures for Visual Recognition and Description

In recent times, digital media contents are inherently of multimedia type, consisting of the form text, audio, image and video. Several of the outstanding computer Vision (CV) problems are being successfully solved with the help of modern Machine Learning (ML) techniques. Plenty of research work has already been carried out in the field of Automatic Image Annotation (AIA), Image Captioning and Video Tagging. Video Captioning, i.e., automatic description generation from digital video, however, is a different and complex problem altogether. This study compares various existing video captioning approaches available today and attempts their classification and analysis based on different parameters, viz., type of captioning methods (generation/retrieval), type of learning models employed, the desired output description length generated, etc. This dissertation also attempts to critically analyze the existing benchmark datasets used in various video captioning models and the evaluation metrics for assessing the final quality of the resultant video descriptions generated. A detailed study of important existing models, highlighting their comparative advantages as well as disadvantages are also included. In this study a novel approach for video captioning on the Microsoft Video Description (MSVD) dataset and Microsoft Video-to-Text (MSR-VTT) dataset is proposed using supervised learning techniques to train a deep combinational framework, for achieving better quality video captioning via predicting semantic tags. We develop simple shallow CNN (2D and 3D) as feature extractors, Deep Neural Networks (DNNs and Bidirectional LSTMs (BiLSTMs) as tag prediction models and Recurrent Neural Networks (RNNs) (LSTM) model as the language model. The aim of the work was to provide an alternative narrative to generating captions from videos via semantic tag predictions and deploy simpler shallower deep model architectures with lower memory requirements as solution so that it is not very memory extensive and the developed models prove to be stable and viable options when the scale of the data is increased. This study also successfully employed deep architectures like the Convolutional Neural Network (CNN) for speeding up automation process of hand gesture recognition and classification of the sign languages of the Indian classical dance form, ‘Bharatnatyam’. This hand gesture classification is primarily aimed at 1) building a novel dataset of 2D single hand gestures belonging to 27 classes that were collected from (i) Google search engine (Google images), (ii) YouTube videos (dynamic and with background considered) and (iii) professional artists under staged environment constraints (plain backgrounds). 2) exploring the effectiveness of CNNs for identifying and classifying the single hand gestures by optimizing the hyperparameters, and 3) evaluating the impacts of transfer learning and double transfer learning, which is a novel concept explored for achieving higher classification accuracy

Design of an Offline Handwriting Recognition System Tested on the Bangla and Korean Scripts

This dissertation presents a flexible and robust offline handwriting recognition system which is tested on the Bangla and Korean scripts. Offline handwriting recognition is one of the most challenging and yet to be solved problems in machine learning. While a few popular scripts (like Latin) have received a lot of attention, many other widely used scripts (like Bangla) have seen very little progress. Features such as connectedness and vowels structured as diacritics make it a challenging script to recognize. A simple and robust design for offline recognition is presented which not only works reliably, but also can be used for almost any alphabetic writing system. The framework has been rigorously tested for Bangla and demonstrated how it can be transformed to apply to other scripts through experiments on the Korean script whose two-dimensional arrangement of characters makes it a challenge to recognize. The base of this design is a character spotting network which detects the location of different script elements (such as characters, diacritics) from an unsegmented word image. A transcript is formed from the detected classes based on their corresponding location information. This is the first reported lexicon-free offline recognition system for Bangla and achieves a Character Recognition Accuracy (CRA) of 94.8%. This is also one of the most flexible architectures ever presented. Recognition of Korean was achieved with a 91.2% CRA. Also, a powerful technique of autonomous tagging was developed which can drastically reduce the effort of preparing a dataset for any script. The combination of the character spotting method and the autonomous tagging brings the entire offline recognition problem very close to a singular solution. Additionally, a database named the Boise State Bangla Handwriting Dataset was developed. This is one of the richest offline datasets currently available for Bangla and this has been made publicly accessible to accelerate the research progress. Many other tools were developed and experiments were conducted to more rigorously validate this framework by evaluating the method against external datasets (CMATERdb 1.1.1, Indic Word Dataset and REID2019: Early Indian Printed Documents). Offline handwriting recognition is an extremely promising technology and the outcome of this research moves the field significantly ahead

A Survey on Semantic Processing Techniques

Semantic processing is a fundamental research domain in computational linguistics. In the era of powerful pre-trained language models and large language models, the advancement of research in this domain appears to be decelerating. However, the study of semantics is multi-dimensional in linguistics. The research depth and breadth of computational semantic processing can be largely improved with new technologies. In this survey, we analyzed five semantic processing tasks, e.g., word sense disambiguation, anaphora resolution, named entity recognition, concept extraction, and subjectivity detection. We study relevant theoretical research in these fields, advanced methods, and downstream applications. We connect the surveyed tasks with downstream applications because this may inspire future scholars to fuse these low-level semantic processing tasks with high-level natural language processing tasks. The review of theoretical research may also inspire new tasks and technologies in the semantic processing domain. Finally, we compare the different semantic processing techniques and summarize their technical trends, application trends, and future directions.Comment: Published at Information Fusion, Volume 101, 2024, 101988, ISSN 1566-2535. The equal contribution mark is missed in the published version due to the publication policies. Please contact Prof. Erik Cambria for detail

Project Borderland: A Multi-Sited Curatorial and Anthropological Probing in Selected Parts of India

This theory-practice PhD project combines multi-sited curatorial and anthropological research in selected north-eastern and eastern borderland sites of India. The borderland is a choice for this research due to its manifoldness. Borders, though manmade and historical, often produce ambiguous lines of divide that are amenable to myths and memories, and related animosities and allegiances in a variety of configurations. The abstract borderland is potentially capable of creating different subject positions like citizens, denizens and non-citizens. This is the project of a curator-participant who works in alternating nuanced roles as participant observer, complicit observer, ethnographer and the critical entity to tease out the different aspects of the borderland from complex anthropological interactions. The research process involves three phases in each site. The first two are the study of the territorial issues via theoretical grounding and fieldwork. These lead to the curatorial intervention in the form of workshops that emerge as knowledge producing situations. The idea is to work with a curatorial strategy that emphasises the processual and is interactive and collaborative, with a view to exploring the shared body of knowledge generated at the workshop mise-en-scènes. Hence, the workshops are conceived as interactive and participatory, involving theatre and cartographic activities among others. Also, the ideas, images and concepts culled from hybrid sources during all the phases of research are juxtaposed here to create fields of multiple inflections, bringing different spaces and times together without merging under a singular discipline. The workshops are, thus, events poised at multi-disciplinary crossroads, where the knowledge of the border experiences maximum density. The project is aimed at studying the relational features of the selected sites; examining the emergence and nature of communities, the role of outsidedness in the implicated cultures and the different temporal registers encountered in the anthropological probing into the physical and metaphorical borderland(s) in their micro-social aspects

End-to-End Multilingual Information Retrieval with Massively Large Synthetic Datasets

End-to-end neural networks have revolutionized various fields of artificial intelligence. However, advancements in the field of Cross-Lingual Information Retrieval (CLIR) have been stalled due to the lack of large-scale labeled data. CLIR is a retrieval task in which search queries and candidate documents are in different languages. CLIR can be very useful in some scenarios: for example, a reporter may want to search foreign-language news to obtain different perspectives for her story; an inventor may explore the patents in another country to understand prior art. This dissertation addresses the bottleneck in end-to-end neural CLIR research by synthesizing large-scale CLIR training data and examining techniques that can exploit this in various CLIR tasks. We publicly release the Large-Scale CLIR dataset and CLIRMatrix, two synthetic CLIR datasets covering a large variety of language directions. We explore and evaluate several neural architectures for end-to-end CLIR modeling. Results show that multilingual information retrieval systems trained on these synthetic CLIR datasets are helpful for many language pairs, especially those in low-resource settings. We further show how these systems can be adapted to real-world scenarios

Language and Identity Theories and experiences in lexicography and linguistic policies in a global world

This book was conceived during the closing event of the DiM project, developed within the framework of the Erasmus plus KA204 - Strategic Partnerships for Adult Education programme. Its fourteen chapters intend to offer food for thought on some of the currently most debated questions for linguists in the global village, and are divided into three thematic sections: 1) multilingualism, minority languages and the eternal dichotomy between orality and writing; 2) lexicography and L2 teaching; 3) the role of linguistics in particularly complex multilingual contexts. The book was published thanks to a grant obtained in 2018 by Regione Friuli Venezia Giulia

Proceedings of the 2nd International Workshop on Digital Language Archives: LangArc 2023

Conference proceedings of the 2nd International Workshop on Digital Language Archives held on June 30, 2023 as part of the ACM/IEEE Joint Conference on Digital Libraries 2023. It includes 10 peer-reviewed papers that were presented at the workshop and an introduction from the workshop organizers