Deep Reinforcement Learning for Chatbots Using Clustered Actions and Human-Likeness Rewards

Training chatbots using the reinforcement learning paradigm is challenging due to high-dimensional states, infinite action spaces and the difficulty in specifying the reward function. We address such problems using clustered actions instead of infinite actions, and a simple but promising reward function based on human-likeness scores derived from human-human dialogue data. We train Deep Reinforcement Learning (DRL) agents using chitchat data in raw text—without any manual annotations. Experimental results using different splits of training data report the following. First, that our agents learn reasonable policies in the environments they get familiarised with, but their performance drops substantially when they are exposed to a test set of unseen dialogues. Second, that the choice of sentence embedding size between 100 and 300 dimensions is not significantly different on test data. Third, that our proposed human-likeness rewards are reasonable for training chatbots as long as they use lengthy dialogue histories of ≥10 sentences

Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis

A large part of the expressive speech synthesis literature focuses on learning prosodic representations of the speech signal which are then modeled by a prior distribution during inference. In this paper, we compare different prior architectures at the task of predicting phoneme level prosodic representations extracted with an unsupervised FVAE model. We use both subjective and objective metrics to show that normalizing flow based prior networks can result in more expressive speech at the cost of a slight drop in quality. Furthermore, we show that the synthesized speech has higher variability, for a given text, due to the nature of normalizing flows. We also propose a Dynamical VAE model, that can generate higher quality speech although with decreased expressiveness and variability compared to the flow based models.Comment: Submitted to ICASSP 202

Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis

This paper proposes an Expressive Speech Synthesis model that utilizes token-level latent prosodic variables in order to capture and control utterance-level attributes, such as character acting voice and speaking style. Current works aim to explicitly factorize such fine-grained and utterance-level speech attributes into different representations extracted by modules that operate in the corresponding level. We show that the fine-grained latent space also captures coarse-grained information, which is more evident as the dimension of latent space increases in order to capture diverse prosodic representations. Therefore, a trade-off arises between the diversity of the token-level and utterance-level representations and their disentanglement. We alleviate this issue by first capturing rich speech attributes into a token-level latent space and then, separately train a prior network that given the input text, learns utterance-level representations in order to predict the phoneme-level, posterior latents extracted during the previous step. Both qualitative and quantitative evaluations are used to demonstrate the effectiveness of the proposed approach. Audio samples are available in our demo page.Comment: Submitted to ICASSP 202

Ensemble-Based Deep Reinforcement Learning for Chatbots

Trainable chatbots that exhibit fluent and human-like conversations remain a big challenge in artificial intelligence. Deep Reinforcement Learning (DRL) is promising for addressing this challenge, but its successful application remains an open question. This article describes a novel ensemble-based approach applied to value-based DRL chatbots, which use finite action sets as a form of meaning representation. In our approach, while dialogue actions are derived from sentence clustering, the training datasets in our ensemble are derived from dialogue clustering. The latter aim to induce specialised agents that learn to interact in a particular style. In order to facilitate neural chatbot training using our proposed approach, we assume dialogue data in raw text only – without any manually-labelled data. Experimental results using chitchat data reveal that (1) near human-like dialogue policies can be induced, (2) generalisation to unseen data is a difficult problem, and (3) training an ensemble of chatbot agents is essential for improved performance over using a single agent. In addition to evaluations using held-out data, our results are further supported by a human evaluation that rated dialogues in terms of fluency, engagingness and consistency – which revealed that our proposed dialogue rewards strongly correlate with human judgements

Effects of heat stress on conception in Holstein and Jersey cattle and oocyte maturation in vitro

Korea, located in East Asia in the northern hemisphere, is experiencing severe climate changes. Specifically, the heat stress caused by global warming is negatively affecting the dairy sector, including milk production and reproductive performance, as the major dairy cattle Holstein-Friesian is particularly susceptible to heat stress. Here, we collected artificial insemination and pregnancy data of the Holstein and the Jersey cows from a dairy farm from 2014 to 2021 and analyzed the association between the conception rate and the temperature-humidity index, calculated using the data from the closest official weather station. As the temperature-humidity index threshold increased, the conception rate gradually decreased. However, this decrease was steeper in the Holstein breed than in the Jersey one at a temperature-humidity index threshold of 75. To evaluate the effects of heat stress on the oocyte quality, we examined the nuclear and cytoplasmic maturation of Holstein (n = 158, obtained from six animals) and Jersey oocytes (n = 123, obtained from six animals), obtained by ovum pick-up. There were no differences in the nuclear maturation between the different conditions (heat stress: 40.5°C, non- heat stress: 37.5°C) or breeds, although the Holstein oocytes seemed to have a lower metaphase II development (p = 0.0521) after in vitro maturation under heat stress conditions. However, we found that the Holstein metaphase II oocytes exposed to heat stress presented more reactive oxygen species and a peripheral distribution of the mitochondria, compared to those of the Jersey cattle. Here, we show that weather information from local meteorological stations can be used to calculate the temperature-humidity index threshold at which heat stress influences the conception rate, and that the Jersey cows are more tolerant to heat stress in terms of their conception rate at a temperature-humidity index over 75. The lower fertility of the Holstein cows is likely attributed to impaired cytoplasmic maturation induced by heat stress. Thus, the Jersey cows can be a good breed for the sustainability of dairy farms for addressing climate changes in South Korea, as they are more resistant to hyperthermia

p15INK4b methylation correlates with thrombocytopenia, blast percentage, and survival in myelodysplastic syndromes in a dose dependent manner: Quantitation using pyrosequencing study

We investigated how the quantity of p15INK4b methylation related to International Prognosic Scoring System variables and survival in 74 patients with de novo myelodysplastic syndrome (MDS). Pyrosequencing of 11 consecutive CpG sites of the p15INK4b promotor region was performed, with the extent of CpG cytosine methylation assessed in terms of methylation level (MtL). Patients with >5% bone marrow blasts had higher MtL than patients with = 2 lineages than in patients with either unilineage or no cytopenia (9.8% vs. 4.1%, p = 0.036, respectively). The survival of patients with >7% MtL was worse than patients with <7% MtL (p = 0.031). Heavy p15INK4b methylation in MDS is associated with IPSS predictors of poor prognosis and adverse survival. Crown Copyright (C) 2009 Published by Elsevier Ltd. All rights reserved.Gao W, 2008, CARCINOGENESIS, V29, P1901, DOI 10.1093/carcin/bgn170Nimer SD, 2008, BLOOD, V111, P4841, DOI 10.1182/blood-2007-08-078139Lee ES, 2008, CLIN CANCER RES, V14, P2664, DOI 10.1158/1078-0432.CCR-07-1232Yang R, 2008, BMC CANCER, V8, DOI 10.1186/1471-2407-8-124MUFTI GJ, 2008, CANC CONTROL, V15, pS14Tost J, 2007, NAT PROTOC, V2, P2265, DOI 10.1038/nprot.2007.314Brakensiek K, 2007, CLIN CHEM, V53, P17, DOI 10.1373/clinchem.2007.072629McDermott KM, 2006, PLOS BIOL, V4, P350, DOI 10.1371/journal/pbio.0040051Shaw RJ, 2006, BRIT J CANCER, V94, P561, DOI 10.1038/sj.bjc.6602972Herman JG, 1996, P NATL ACAD SCI USA, V93, P9821Uchida T, 1997, BLOOD, V90, P1403Quesnel B, 1998, BLOOD, V91, P2985Ronaghi M, 1998, SCIENCE, V281, P363Cameron EE, 1999, BLOOD, V94, P2445CZEPULKOWSKI B, 2001, HUMAN CYTOGENETICS M, P1Tien HF, 2001, BRIT J HAEMATOL, V112, P148Uhlmann K, 2002, ELECTROPHORESIS, V23, P4072Au WY, 2003, BRIT J HAEMATOL, V120, P1062THIELE J, 1991, J CLIN PATHOL, V44, P300Colella S, 2003, BIOTECHNIQUES, V35, P146Chim CS, 2003, ANN HEMATOL, V82, P738, DOI 10.1007/s00277-003-0744-8SHAFFER LG, 2005, ISCN 2005 INT SYSTEMMARSH S, 2005, METH MOL B, V311, P97Dupont JM, 2004, ANAL BIOCHEM, V333, P119, DOI 10.1016/j.ab.2004.05.007Cottrell SE, 2004, CLIN BIOCHEM, V37, P595, DOI 10.1016/j.clinbiochem.2004.05.010LIST AF, 2004, AM SOC HEMATOL ED PR, P297