On the Ambiguity of Rank-Based Evaluation of Entity Alignment or Link Prediction Methods

In this work, we take a closer look at the evaluation of two families of methods for enriching information from knowledge graphs: Link Prediction and Entity Alignment. In the current experimental setting, multiple different scores are employed to assess different aspects of model performance. We analyze the informativeness of these evaluation measures and identify several shortcomings. In particular, we demonstrate that all existing scores can hardly be used to compare results across different datasets. Moreover, we demonstrate that varying size of the test size automatically has impact on the performance of the same model based on commonly used metrics for the Entity Alignment task. We show that this leads to various problems in the interpretation of results, which may support misleading conclusions. Therefore, we propose adjustments to the evaluation and demonstrate empirically how this supports a fair, comparable, and interpretable assessment of model performance. Our code is available at https://github.com/mberr/rank-based-evaluation

PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings

Recently, knowledge graph embeddings (KGEs) received significant attention, and several software libraries have been developed for training and evaluating KGEs. While each of them addresses specific needs, we re-designed and re-implemented PyKEEN, one of the first KGE libraries, in a community effort. PyKEEN 1.0 enables users to compose knowledge graph embedding models (KGEMs) based on a wide range of interaction models, training approaches, loss functions, and permits the explicit modeling of inverse relations. Besides, an automatic memory optimization has been realized in order to exploit the provided hardware optimally, and through the integration of Optuna extensive hyper-parameter optimization (HPO) functionalities are provided

Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework

The heterogeneity in recently published knowledge graph embedding models' implementations, training, and evaluation has made fair and thorough comparisons difficult. In order to assess the reproducibility of previously published results, we re-implemented and evaluated 21 interaction models in the PyKEEN software package. Here, we outline which results could be reproduced with their reported hyper-parameters, which could only be reproduced with alternate hyper-parameters, and which could not be reproduced at all as well as provide insight as to why this might be the case. We then performed a large-scale benchmarking on four datasets with several thousands of experiments and 24,804 GPU hours of computation time. We present insights gained as to best practices, best configurations for each model, and where improvements could be made over previously published best configurations. Our results highlight that the combination of model architecture, training approach, loss function, and the explicit modeling of inverse relations is crucial for a model's performances, and not only determined by the model architecture. We provide evidence that several architectures can obtain results competitive to the state-of-the-art when configured carefully. We have made all code, experimental configurations, results, and analyses that lead to our interpretations available at https://github.com/pykeen/pykeen and https://github.com/pykeen/benchmarkin

Assessment of the impact of climate change in temperate zone on grain legume yield and N2 fixation

International audienceClimate change is likely to strengthen abiotic stresses on crops in temperate zones. Grain legumes and the associated provision of ecosystem services are the cornerstone of more sustainable cropping systems, yet the impact of climate change on their performance has not been extensively quantified. Based on previous experiments carried out in south-western France with low biotic stress, we calibrated the STICS soil-crop model for spring pea (SP), winter pea (WP) and winter faba bean (WF) and evaluated its quality of prediction on an independent dataset. STICS was used to explore the effect of climate change scenarios on the legumes performance. Assuming no change in crop management, mean and inter-annual variability of grain yield and N2 fixation were assessed for historical (1995-2015), mid-term (2020-2040) and long-term (2060-2080) periods, considering projections from two Global Circulating Models (GCM) and two Representative CO2 Concentration Pathways (RCP), i.e. RCP 4.5 and RCP 8.5. The GCMs consistently predicted no significant change in rainfall amounts and patterns but indicated a 1.7°C and 2.5°C increase in average temperature over the growth period in the long term under RCP 4.5 and RCP 8.5 respectively. Therefore, simulations indicated no extra water stress on grain yield and N2 fixation of these legumes. The increase in temperature entailed a shortening in crop duration and a slight but significant increase in the temperature stress factor values for grain filling, for photosynthesis and for N2 fixation during the reproductive period (+1% to +13 % depending on temperature stress, crop and RCP). Under RCP 4.5, yield decreased by 23 to 34% (depending on crop) in the long term. Average fixed N2 decreased by 16% to 34%. Probability of yield failure (i.e. yield below the 20th percentile of historical yield) increased from 20 to 50, 54 and 58% for WF, WP and SP respectively. Probability of N2 fixation failure increased from 20 to 34, 50 and 53% for WP, WF and SP respectively. In contrast, under RCP 8.5, the CO2 fertilisation effect would offset the decrease in yield due to the increase in temperature and simulations predicted a 8 to 13 % average yield increase in the long term. Average N2 fixation would benefit from the increase in biomass and increase by 15 to 23%. Probability of yield failure would increase slightly, from 20 to 21, 25 and 27% for WF, WP and SP respectively. Probability of N2 fixation failure would increase for spring pea (from 20 to 31%) but decrease for winter faba bean (from 20 to 13%) and winter pea (from 20 to 11%). The increased probability of yield and N2 fixation failure simulated with the RCP 4.5 scenario indicates the need for technical and transformational adaptations for grain legumes to deliver the expected ecosystem services with future climate. Under RCP 8.5, better yield and N2 fixation highlight the opportunity represented by climate change for inclusion of grain legume in cropping systems

Contrasted response to climate change of winter and spring grain legumes in southwestern France

International audienceClimate change could undermine grain legumes ability to fix atmospheric nitrogen and their contribution to increase cropping systems sustainability. Pea (Pisum sativum L.) and faba bean (Vicia faba L.) are the two most widely grown grain legumes in Europe, yet the potential impact of climate change on their performances has not been quantified. We calibrated and evaluated the STICS soil-crop model for spring pea, winter pea and winter faba bean using experimental data from southwestern France and explored the effect of contrasting climate change scenarios. After calibration, STICS accurately simulated grain yield and amount of N2 fixed for the experimental growing seasons. Assuming no change in crop management, mean and inter-annual variability of grain yield and fixed N2 were assessed for historical (1995−2015), mid-term (2020−2040) and long-term (2060−2080) periods in one location in southwestern France. We considered projections from three climate models and two Representative [CO2] Pathways (RCP 4.5 and RCP 8.5). The climate models spanned a wide range of changes in temperature (+0.3 to +4.1 °C) and rainfall (−15% to +8%) depending on time horizon and RCP. Simulated grain yield increased over the long term in most scenarios (+1 to +25%), and spring pea tended to benefit less than winter pea and winter faba bean. Nevertheless, for the climate scenario with a decrease in rainfall and the strongest increase in temperature, simulated spring pea grain yield decreased by 28% while winter legumes yields were less affected (−14% for pea and no decrease for faba bean). Simulated changes in the amount of N2 fixed followed the grain yield response. Temperature rise caused a shortening in crop cycle duration. Simulated temperature stress significantly increased for spring and winter pea in most climate change scenarios while winter faba bean was rather unaffected due to greater upper temperature thresholds. N2 fixation of spring pea was reduced by above-optimal temperature during its vegetative growth in spring while N2 fixation of winter legumes was enhanced by the increase in temperature during their vegetative growth in winter. Simulated drought stress only increased in the climate scenario predicting a decrease in rainfall. Overall, [CO2] increase would allow offsetting negative effects of temperature and drought on grain yield and N2 fixation, except for climate scenarios involving a decrease in rainfall and the strong increase in temperature. The contrasted simulated response of winter and spring grain legumes to climate change in southwestern France points to the opportunity to tap grain legume diversity and cultivar choice as an adaptation strategy