Parameter-efficient modeling and robust automatic evaluation of image captioning

Le sous-titrage d’images est la tâche de l’intelligence artificielle (IA) qui consiste à décrire des images en langage naturel. Cette tâche d’IA a plusieurs applications sociétales utiles, telles que l’accessibilité pour les malvoyants, la génération automatisée de contenu, l’interaction humain-robot et l’analyse d’imagerie médicale. Au cours des huit dernières années, la recherche sur le sous-titrage d'images a connu d'énormes progrès dans la création de modèles solides, la collecte d'ensembles de données à grande échelle ainsi que le développement de mesures d'évaluation automatique. Malgré ces progrès remarquables, la recherche sur le sous-titrage d'images est confrontée à deux défis majeurs: 1) Comment construire des modèles efficaces en termes de paramètres, et 2) Comment construire des métriques d'évaluation automatique robustes. Dans cette thèse, nous apportons notre contribution à la résolution de chacun de ces défis. Premièrement, nous proposons une méthode efficace en termes de paramètres (MAPL \cite{mapl}) qui adapte des modèles pré-entraînés unimodaux de vision uniquement et de langage uniquement pour la tâche multimodale de sous-titrage d'images. MAPL apprend un mappage léger entre les espaces de représentation des modèles unimodaux. Ainsi, MAPL peut exploiter les fortes capacités de généralisation des modèles unimodaux pré-entraînés pour des tâches multimodales telles que le sous-titrage d'images. Deuxièmement, nous présentons une étude systématique de la robustesse des mesures d’évaluation des sous-titres d’images récemment proposées. Même si ces métriques correspondent bien aux jugements humains, nous avons constaté qu'elles ne sont pas robustes pour identifier les erreurs fines dans les légendes générées par le modèle. Il faut donc faire preuve de prudence lors de l'utilisation de ces métriques pour l'évaluation des sous-titres d'images. Nous espérons que nos résultats guideront de nouvelles améliorations dans l’évaluation automatique du sous-titrage d’images.Image captioning is the artificial intelligence (AI) task of describing images in natural language. This AI task has several useful societal applications, such as accessibility for the visually impaired, automated content generation, human-robot interaction, and medical imaging analysis. Over the last eight years, image captioning research has seen tremendous progress in building strong models, collecting large scale datasets as well as developing automatic evaluation metrics. Despite such remarkable progress, image captioning research faces two major challenges: 1) How to build parameter-efficient models, and 2) How to build robust automatic evaluation metrics. In this thesis, we make contributions towards tackling each of these challenges. First, we propose a parameter efficient method (MAPL \cite{mapl}) that adapts pre-trained unimodal vision-only and language-only models for the multimodal task of image captioning. MAPL learns a lightweight mapping between the representation spaces of the unimodal models. Thus, MAPL can leverage the strong generalization capabilities of the pre-trained unimodal models for multimodal tasks such as image captioning. Second, we present a systematic study of the robustness of recently proposed image captioning evaluation metrics. Even though these metrics correlate well with human judgments, we found that these metrics are not robust in identifying fine-grained errors in model generated captions, and thus, caution needs to be exercised when using these metrics for image captioning evaluation. We hope our findings will guide further improvements in the automatic evaluation of image captioning

An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics

Recently, reference-free metrics such as CLIPScore (Hessel et al., 2021) and UMIC (Lee et al., 2021) have been proposed for automatic evaluation of image captions, demonstrating a high correlation with human judgment. In this work, our focus lies in evaluating the robustness of these metrics in scenarios that require distinguishing between two captions with high lexical overlap but very different meanings. Our findings reveal that despite their high correlation with human judgment, both CLIPScore and UMIC struggle to identify fine-grained errors in captions. However, when comparing different types of fine-grained errors, both metrics exhibit limited sensitivity to implausibility of captions and strong sensitivity to lack of sufficient visual grounding. Probing further into the visual grounding aspect, we found that both CLIPScore and UMIC are impacted by the size of image-relevant objects mentioned in the caption, and that CLIPScore is also sensitive to the number of mentions of image-relevant objects in the caption. In terms of linguistic aspects of a caption, we found that both metrics lack the ability to comprehend negation, UMIC is sensitive to caption lengths, and CLIPScore is insensitive to the structure of the sentence. We hope our findings will serve as a valuable guide towards improving reference-free evaluation in image captioning

Full and Modified Glasgow-Blatchford Bleeding Score in Predicting the Outcome of Patients with Acute Upper Gastrointestinal Bleeding; a Diagnostic Accuracy Study

Introduction: Screening of high risk patients and accelerating their treatment measures can reduce the burden of the disease caused by acute upper gastrointestinal (GI) bleeding. This study aimed to compare the full and modified Glasgow-Blatchford Bleeding Score (GBS and mGBS) in prediction of in-hospital outcomes of upper GI bleeding.Methods: In the present retrospective cross-sectional study, the accuracy of GBS and mGBS models were compared in predicting the outcome of patients over 18 years of age with acute upper GI bleeding confirmed via endoscopy, presenting to the emergency departments of 3 teaching hospitals during 4 years.Results: 330 cases with the mean age of 59.07 ± 19.00 years entered the study (63.60% male). Area under the curve of GBS and mGBS scoring systems were 0.691 and 0.703, respectively, in prediction of re-bleeding (p = 0.219), 0.562 and 0.563 regarding need for surgery (p = 0.978), 0.549 and 0.542 for endoscopic intervention (p = 0.505), and 0.767 and 0.770 regarding blood transfusion (p = 0.753). Area under the ROC curve of GBS scoring system regarding need for hospitalization in intensive care unit (0.589 vs. 0.563; p = 0.035) and mortality (0.597 vs. 0.564; p = 0.011) was better but the superiority was not clinically significant.Conclusion: GBS and mGBS scoring systems have similar accuracy in prediction of the probability of re-bleeding, need for blood transfusion, surgery and endoscopic intervention, hospitalization in intensive care unit, and mortality of patients with acute upper GI bleeding

Design, synthesis and anti platelet aggregation studies of new α -phenyl cinnamonitrile derivatives

Introduction:Cardiovascular and thromboembolic diseases are one of the most common causes of death in the world. Platelets play an important role in the pathogenesis of cardiovascular diseases. The use of antiplatelet drugs is one of the most important ways of prevention and treatment cardiovascular disorders. Looking at the various complications of the anti platelet drugs of the old generation, researchers are always looking for new drugs in the field of anti platelet therapy. There are some reports indicating that α-phenyl cinnamonitrile can be useful for treatment of cardiovascular diseases.  Therefore this study is designed to explore the anti platelet activity of a selected group of α-phenyl cinnamonitriles. Methods and Results:Benzyl cyanide and various benzaldehyde derivatives were reacted in 90% ethanol to obtain the title compounds. A solution of sodium methoxide in methanol was added to this mixture dropwise, with stirring.. The product, thus obtained, was filtered off and crystallized from proper solvent. In the next step using hydrolysis of the synthesized derivatives in the presence of TFA, acetic acid and98% sulfuric acid, amide derivatives of the nitrile compounds were synthesized. The structure of the synthesized compounds was confirmed by using NMR, IR, and MS spectrometry methods. Invitro anti platelet activity of α-phenyl cinnamonitriles  and  their amide congeners were evaluated by using arachidonic acid (AA) and adenosine diphosphate (ADP) as inducer according to born method on human platelet rich plasma (PRP). For all compounds IC50s were calculated and compared. The result showed that α-phenyl cinnamonitrile, 4- methoxy-α-phenyl cinnamonitrile, were the most potent anti platelet aggregation agents with IC50 values of 17.79 and 38.2 µM respectively. R=2-Cl; 2-Br; 2-OH; 3-F;3-Cl;3-Br;3-OH; 4-H; 4-OMe;4-F; 4-Cl; 4-CN; 4-Br; 3,4-diOMe ;3, 4, 5-triOMe. Conclusions:A group of α-β unsaturated nitrile derivative is introduced as new anti platelet aggregation agents proving the eligibility of this scaffold for anti platelet activity

Social Aspects of Algorithms: Fairness, Diversity, and Resilience to Strategic Behavior

With algorithms becoming ubiquitous in our society, it is important to ensure that they are compatible with our social values. In this thesis, we study some of the social aspects of algorithms including fairness, diversity, and resilience to strategic behavior of individuals. Lack of diversity has a potential impact on discrimination against marginalized groups. Inspired by this issue, in the first part of this thesis, we study a notion of diversity in bipartite matching problems. Bipartite matching where agents on one side of a market are matched to one or more agents or items on the other side, is a classical model that is used in myriad application areas such as healthcare, advertising, education, and general resource allocation. In particular, we consider an application of matchings where a firm wants to hire, i.e. match, some workers for a number of teams. Each team has a demand that needs to be satisfied, and each worker has multiple features (e.g., country of origin, gender). We ask the question of how to assign workers to the teams in an efficient way, i.e. low-cost matching, while forming diverse teams with respect to all the features. Inspired by previous work, we balance whole-match diversity and economic efficiency by optimizing a supermodular function over the matching. Particularly, we show when the number of features is given as part of the input, this problem is NP-hard, and design a pseudo-polynomial time algorithm to solve this problem. Next, we focus on studying fairness in optimization problems. Particularly, in this thesis, we study two notions of fairness in an optimization problem called correlation clustering. In correlation clustering, given an edge-weighted graph, each edge in addition to a weight has a positive or negative label. The goal is to obtain a clustering of the vertices into an arbitrary number of clusters that minimizes disagreements which is defined as the total weight of negative edges trapped inside a cluster plus the sum of weights of positive edges between different clusters. In the first fairness notion, assuming each node has a color, i.e. feature, our aim is to generate clusters with minimum disagreements, where the distribution of colors in each cluster is the same as the global distribution. Next, we switch our attention to a min-max notion of fairness in correlation clustering. In this notion of fairness, we consider a cluster-wise objective function that asks to minimize the maximum number of disagreements of each cluster. In this notion, the goal is to respect the quality of each cluster. We focus on designing approximation algorithms for both of these notions. In the last part of this thesis, we take into consideration, the vulnerability of algorithms to manipulation and gaming. We study the problem of how to learn a linear classifier in presence of strategic agents that desire to be classified as positive and that are able to modify their position by a limited amount, making the classifier not be able to observe the true position of agents but rather a position where the agent pretends to be. We focus on designing algorithms with a bounded number of mistakes for a few different variations of this problem

Setting Fair Incentives to Maximize Improvement

We consider the problem of helping agents improve by setting goals. Given a set of target skill levels, we assume each agent will try to improve from their initial skill level to the closest target level within reach (or do nothing if no target level is within reach). We consider two models: the common improvement capacity model, where agents have the same limit on how much they can improve, and the individualized improvement capacity model, where agents have individualized limits. Our goal is to optimize the target levels for social welfare and fairness objectives, where social welfare is defined as the total amount of improvement, and we consider fairness objectives when the agents belong to different underlying populations. We prove algorithmic, learning, and structural results for each model. A key technical challenge of this problem is the non-monotonicity of social welfare in the set of target levels, i.e., adding a new target level may decrease the total amount of improvement; agents who previously tried hard to reach a distant target now have a closer target to reach and hence improve less. This especially presents a challenge when considering multiple groups because optimizing target levels in isolation for each group and outputting the union may result in arbitrarily low improvement for a group, failing the fairness objective. Considering these properties, we provide algorithms for optimal and near-optimal improvement for both social welfare and fairness objectives. These algorithmic results work for both the common and individualized improvement capacity models. Furthermore, despite the non-monotonicity property and interference of the target levels, we show a placement of target levels exists that is approximately optimal for the social welfare of each group. Unlike the algorithmic results, this structural statement only holds in the common improvement capacity model, and we illustrate counterexamples of this result in the individualized improvement capacity model. Finally, we extend our algorithms to learning settings where we have only sample access to the initial skill levels of agents

The Role Of Indigenous Knowledge In Increasing Rural People Knowledge About Agriculture 1

ABSTRACT Our today's world is the contradictions and collision's world. Contradiction between cultures, religions, different societies and countries. In recently years, from Renaissance till now, as much as human had developed, they also had contradictions and collisions in their world . One of these contradictions is the contrast between tradition and modernism. Maybe we can find these contrast roots in colonial era, the time when colonists promote their innovation in their colonies. Mostly these techniques and innovations show their native knowledge and the way of their living is foolish and inefficient and tried to enter industrial ways in to their life to increase production efficiency through this way. Thus the way of their living which was been formed during thousands of years has gone to be forgotten little by little. We can say, agriculture part is bearing the most damage in this rapid industrialization process. Absolving old and compatible ways in agriculture part and replacing and using of implant, harvest patterns without any proportions with environment has caused decrease of production efficiency, soil erosion and hard destruction of environment during a long time. Finally, at the end of the 20 th century decades, some solutions were suggested to solve these inconsistencies and problems. So the importance of native knowledge and effort in compilation of that with modern knowledge were considered and it was tried to make general and stable view in relation with environment and the way of living through this way