Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by individual attention heads in the encoder to the overall performance of the model and analyze the roles played by them. We find that the most important and confident heads play consistent and often linguistically-interpretable roles. When pruning heads using a method based on stochastic gates and a differentiable relaxation of the L0 penalty, we observe that specialized heads are last to be pruned. Our novel pruning method removes the vast majority of heads without seriously affecting performance. For example, on the English-Russian WMT dataset, pruning 38 out of 48 encoder heads results in a drop of only 0.15 BLEU.Comment: ACL 2019 (camera-ready

Context-Aware Monolingual Repair for Neural Machine Translation

Modern sentence-level NMT systems often produce plausible translations of isolated sentences. However, when put in context, these translations may end up being inconsistent with each other. We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations. DocRepair performs automatic post-editing on a sequence of sentence-level translations, refining translations of sentences in context of each other. For training, the DocRepair model requires only monolingual document-level data in the target language. It is trained as a monolingual sequence-to-sequence model that maps inconsistent groups of sentences into consistent ones. The consistent groups come from the original training data; the inconsistent groups are obtained by sampling round-trip translations for each isolated sentence. We show that this approach successfully imitates inconsistencies we aim to fix: using contrastive evaluation, we show large improvements in the translation of several contextual phenomena in an English-Russian translation task, as well as improvements in the BLEU score. We also conduct a human evaluation and show a strong preference of the annotators to corrected translations over the baseline ones. Moreover, we analyze which discourse phenomena are hard to capture using monolingual data only

Improvement of self-organization, relations and learning motivation of different age students by integrative teaching methodology

An integrative teaching methodology was developed by qualified specialists and implemented in 17 Latvian schools during one study year. Students who participated in the learning of integrative teaching methodology were divided into 3 age groups: 3–4; 5–7 and 8–12 grade students. Students and teachers who participated in the teaching of integrative teaching methodology were questionnaire before and after using of this methodology. The questionnaires about self-organization problems, relations between students and behaviour of students were filled by teachers but the questionnaire about the learning motivation of students was filled by students themselves. The results of questionnaire showed that problems of self-organization, motivation, hyperactivity, attention deficiency, social anxiety, communication and depressive symptomatic of students who were taught according to the integrative teaching methodology of general schools have reduced in all age groups but in 5–7 and 8–12 grade groups of student's aggressive behaviour reduced as well. Self-organization and behaviour problems of students from special schools after realizing integrative teaching methodology have less reduced in comparison with students of general education schools. In general education schools as well as in special schools external learning motivation and internal learning motivation of all age students increased but am motivation or lack of motivation to learn decreased

Інтракраніальні судинні ускладнення при СНІДі

The article analyzed intracranial pathomorphological changes in HIV patients as well as describes vessel wall changes leading to ischemic or hemorrhagic stroke. There are considered etiological factors for ischemic and hemorrhagic stroke in HIV patients. The article describes varied pathological processes and morphological changes in brain vessels which are called HIV-associated cerebral vasculopathy. Radiologically HIV-associated cerebral vasculopathy is characterized by fusiform aneurysms, dolichoectasia, artery stenosis or occlusion. Stroke clinical manifestation is the same for patients with HIV as for the other population, although the average age in HIV patients is much younger. The principles for stroke therapy are the same in HIV patients as in general population. Drug and surgical treatment strategies are both used. The most common surgical method is endovascular technique. Anti-stroke therapy is recommended to be completed with antiretroviral therapy being continued.Представлены интракраниальные патоморфологические осложнения у пациентов с вирусом иммунодефицита человека (ВИЧ). Рассмотрены изменения сосудистой стенки, лежащие в основе нарушений мозгового кровообращения как ишемического, так и геморрагического характера. Приведены этиологические факторы ишемических и геморрагических инсультов у ВИЧ-инфицированных. Освещены патологические процессы и морфологическая основа изменений в стенке церебральных сосудов – “ВИЧ-ассоциированная церебральная васкулопатия”. Радиологическими признаками ВИЧ-ассоциированной церебральной васкулопатии являются: фузиформные аневризмы, долихоэктазия, стеноз или окклюзия пораженного сосуда. Клиника инсультов у пациентов с ВИЧ не отличается от таковой в общей популяции, однако средний возраст у ВИЧ- инфицированных гораздо меньше. Принципы лечения инсультов у пациентов с ВИЧ не отличаются от общепризнанных. Применяют консервативное и хирургическое лечение. Из методов хирургического лечения предпочтение отдают эндоваскулярной технике. Во время лечения инсульта рекомендуют продолжать антиретровирусную терапию.Представлено інтракраніальні патоморфологічні ускладнення у пацієнтів з вірусом імунодефициту людини (ВІЛ). Розглянуто зміни судинної стінки, які лежать в основі порушень мозкового кровообігу як ішемічного, так і геморагічного характеру. Наведено етіологічні чинники ішемічних і геморагічних інсультів у ВІЛ-інфікованих. Висвітлено патологічні процеси та морфологічну основу змін у стінці церебральних судин – “ВІЛ-асоційовану церебральну васкулопатію”. Радіологічними ознаками ВІЛ-асоційованої церебральної васкулопатії є: фузіформні аневризми, доліхоектазія, стеноз або оклюзія ураженої судини. Клініка інсультів у пацієнтів з ВІЛ не відрізняється від такої в загальній популяції, проте середній вік у ВІЛ-інфікованих набагато менше. Принципи лікування інсультів у пацієнтів з ВІЛ не відрізняються від загальновизнаних. Застосовують консервативне і хірургічне лікування. З методів хірургічного лікування перевагу віддають ендоваскулярній техніці. Під час лікування інсульту рекомендують продовжувати антиретровірусну терапію.