From feature to paradigm: deep learning in machine translation

In the last years, deep learning algorithms have highly revolutionized several areas including speech, image and natural language processing. The specific field of Machine Translation (MT) has not remained invariant. Integration of deep learning in MT varies from re-modeling existing features into standard statistical systems to the development of a new architecture. Among the different neural networks, research works use feed- forward neural networks, recurrent neural networks and the encoder-decoder schema. These architectures are able to tackle challenges as having low-resources or morphology variations. This manuscript focuses on describing how these neural networks have been integrated to enhance different aspects and models from statistical MT, including language modeling, word alignment, translation, reordering, and rescoring. Then, we report the new neural MT approach together with a description of the foundational related works and recent approaches on using subword, characters and training with multilingual languages, among others. Finally, we include an analysis of the corresponding challenges and future work in using deep learning in MTPostprint (author's final draft

The Roles of Language Models and Hierarchical Models in Neural Sequence-to-Sequence Prediction

With the advent of deep learning, research in many areas of machine learning is converging towards the same set of methods and models. For example, long short-term memory networks are not only popular for various tasks in natural language processing (NLP) such as speech recognition, machine translation, handwriting recognition, syntactic parsing, etc., but they are also applicable to seemingly unrelated fields such as robot control, time series prediction, and bioinformatics. Recent advances in contextual word embeddings like BERT boast with achieving state-of-the-art results on 11 NLP tasks with the same model. Before deep learning, a speech recognizer and a syntactic parser used to have little in common as systems were much more tailored towards the task at hand. At the core of this development is the tendency to view each task as yet another data mapping problem, neglecting the particular characteristics and (soft) requirements tasks often have in practice. This often goes along with a sharp break of deep learning methods with previous research in the specific area. This work can be understood as an antithesis to this paradigm. We show how traditional symbolic statistical machine translation models can still improve neural machine translation (NMT) while reducing the risk for common pathologies of NMT such as hallucinations and neologisms. Other external symbolic models such as spell checkers and morphology databases help neural grammatical error correction. We also focus on language models that often do not play a role in vanilla end-to-end approaches and apply them in different ways to word reordering, grammatical error correction, low-resource NMT, and document-level NMT. Finally, we demonstrate the benefit of hierarchical models in sequence-to-sequence prediction. Hand-engineered covering grammars are effective in preventing catastrophic errors in neural text normalization systems. Our operation sequence model for interpretable NMT represents translation as a series of actions that modify the translation state, and can also be seen as derivation in a formal grammar.EPSRC grant EP/L027623/1 EPSRC Tier-2 capital grant EP/P020259/

NEW ADVANCES IN REORDERING FOR STATISTICAL MACHINE TRANSLATION

Ph.DDOCTOR OF PHILOSOPH

Prediction of Quality of Experience for Video Streaming Using Raw QoS Parameters

Along with the rapid growth in consumer adoption of modern portable devices, video streaming is expected to dominate a large share of the global Internet traffic in the near future. Today user experience is becoming a reliable indicator for video service providers and telecommunication operators to convey overall end-to-end system functioning. Towards this, there is a profound need for an efficient Quality of Experience (QoE) monitoring and prediction. QoE is a subjective metric, which deals with user perception and can vary due to the user expectation and context. However, available QoE measurement techniques that adopt a full reference method are impractical in real-time transmission since they require the original video sequence to be available at the receiver’s end. QoE prediction, however, requires a firm understanding of those Quality of Service (QoS) factors that are the most influential on QoE. The main aim of this thesis work is the development of novel and efficient models for video quality prediction in a non-intrusive way and to demonstrate their application in QoE-enabled optimisation schemes for video delivery. In this thesis, the correlation between QoS and QoE is utilized to objectively estimate the QoE. For this, both objective and subjective methods were used to create datasets that represent the correlation between QoS parameters and measured QoE. Firstly, the impact of selected QoS parameters from both encoding and network levels on video QoE is investigated. The obtained QoS/QoE correlation is backed by thorough statistical analysis. Secondly, the development of two novel hybrid non-reference models for predicting video quality using fuzzy logic inference systems (FIS) as a learning-based technique. Finally, attention was move onto demonstrating two applications of the developed FIS prediction model to show how QoE is used to optimise video delivery

Contribution to reliable end-to-end communication over 5G networks using advanced techniques

5G cellular communication, especially with its hugely available bandwidth provided by millimeter-wave, is a promising technology to fulfill the coming high demand for vast data rates. These networks can support new use cases such as Vehicle to Vehicle and augmented reality due to its novel features such as network slicing along with the mmWave multi-gigabit-persecond data rate. Nevertheless, 5G cellular networks suffer from some shortcomings, especially in high frequencies because of the intermittent nature of channels when the frequency rises. Non-line of sight state is one of the significant issues that the new generation encounters. This drawback is because of the intense susceptibility of higher frequencies to blockage caused by obstacles and misalignment. This unique characteristic can impair the performance of the reliable transport layer widely deployed protocol, TCP, in attaining high throughput and low latency throughout a fair network. As a result, the protocol needs to adjust the congestion window size based on the current situation of the network. However, TCP cannot adjust its congestion window efficiently, which leads to throughput degradation of the protocol. This thesis presents a comprehensive analysis of reliable end-to-end communications in 5G networks and analyzes TCP’s behavior in one of the 3GPP’s well-known s cenarios called urban deployment. Furtherm ore, two novel TCPs bas ed on artificial intelligence have been proposed to deal with this issue. The first protocol uses Fuzzy logic, a subset of artificial intelligence, and the second one is based on deep learning. The extensively conducted simulations showed that the newly proposed protocols could attain higher performance than common TCPs, such as BBR, HighSpeed, Cubic, and NewReno in terms of throughput, RTT, and sending rate adjustment in the urban scenario. The new protocols' superiority is achieved by employing smartness in the conges tions control mechanism of TCP, which is a powerful enabler in fos tering TCP’s functionality. To s um up, the 5G network is a promising telecommunication infrastructure that will revolute various aspects of communication. However, different parts of the Internet, such as its regulations and protocol stack, will face new challenges, which need to be solved in order to exploit 5G capacity, and without intelligent rules and protocols, the high bandwidth of 5G, especially 5G mmWave will be wasted. Two novel schemes to solve the issues have been proposed based on an Artificial Intelligence subset technique called fuzzy and a machine learning-based approach called Deep learning to enhance the performance of 5G mmWave by improving the functionality of the transport layer. The obtained results indicated that the new schemes could improve the functionality of TCP by giving intelligence to the protocol. As the protocol works more smartly, it can make sufficient decisions on different conditions.La comunicació cel·lular 5G, especialment amb l’amplada de banda molt disponible que proporciona l’ona mil·limètrica, és una tecnologia prometedora per satisfer l’elevada demanda de grans velocitats de dades. Aquestes xarxes poden admetre casos d’ús nous, com ara Vehicle to Vehicle i realitat augmentada, a causa de les seves novetats, com ara el tall de xarxa juntament amb la velocitat de dades mWave de multi-gigabit per segon. Tot i això, les xarxes cel·lulars 5G pateixen algunes deficiències, sobretot en freqüències altes a causa de la naturalesa intermitent dels canals quan augmenta la freqüència. L’estat de no visió és un dels problemes significatius que troba la nova generació. Aquest inconvenient es deu a la intensa susceptibilitat de freqüències més altes al bloqueig causat per obstacles i desalineació. Aquesta característica única pot perjudicar el rendiment del protocol TCP, àmpliament desplegat, de capa de transport fiable en aconseguir un alt rendiment i una latència baixa en tota una xarxa justa. Com a resultat, el protocol ha d’ajustar la mida de la finestra de congestió en funció de la situació actual de la xarxa. Tot i això, TCP no pot ajustar la seva finestra de congestió de manera eficient, cosa que provoca una degradació del rendiment del protocol. Aquesta tesi presenta una anàlisi completa de comunicacions extrem a extrem en xarxes 5G i analitza el comportament de TCP en un dels escenaris coneguts del 3GPP anomenat desplegament urbà. A més, s'han proposat dos TCP nous basats en intel·ligència artificial per tractar aquest tema. El primer protocol utilitza la lògica Fuzzy, un subconjunt d’intel·ligència artificial, i el segon es basa en l’aprenentatge profund. Les simulacions àmpliament realitzades van mostrar que els protocols proposats recentment podrien assolir un rendiment superior als TCP habituals, com ara BBR, HighSpeed, Cubic i NewReno, en termes de rendiment, RTT i ajust d’índex d’enviament en l’escenari urbà. La superioritat dels nous protocols s’aconsegueix utilitzant la intel·ligència en el mecanisme de control de congestions de TCP, que és un poderós facilitador per fomentar la funcionalitat de TCP. En resum, la xarxa 5G és una prometedora infraestructura de telecomunicacions que revolucionarà diversos aspectes de la comunicació. No obstant això, diferents parts d’Internet, com ara les seves regulacions i la seva pila de protocols, s’enfrontaran a nous reptes, que cal resoldre per explotar la capacitat 5G, i sens regles i protocols intel·ligents, l’amplada de banda elevada de 5G, especialment 5G mmWave, pot ser desaprofitat. S'han proposat dos nous es quemes per resoldre els problemes basats en una tècnica de subconjunt d'Intel·ligència Artificial anomenada “difusa” i un enfocament basat en l'aprenentatge automàtic anomenat “Aprenentatge profund” per millorar el rendiment de 5G mmWave, millorant la funcionalitat de la capa de transport. Els resultats obtinguts van indicar que els nous esquemes podrien millorar la funcionalitat de TCP donant intel·ligència al protocol. Com que el protocol funciona de manera més intel·ligent, pot prendre decisions suficients en diferents condicionsPostprint (published version

An Object-Oriented Algorithmic Laboratory for Ordering Sparse Matrices

We focus on two known NP-hard problems that have applications in sparse matrix computations: the envelope/wavefront reduction problem and the fill reduction problem. Envelope/wavefront reducing orderings have a wide range of applications including profile and frontal solvers, incomplete factorization preconditioning, graph reordering for cache performance, gene sequencing, and spatial databases. Fill reducing orderings are generally limited to—but an inextricable part of—sparse matrix factorization. Our major contribution to this field is the design of new and improved heuristics for these NP-hard problems and their efficient implementation in a robust, cross-platform, object-oriented software package. In this body of research, we (1) examine current ordering algorithms, analyze their asymptotic complexity, and characterize their behavior in model problems, (2) introduce new and improved algorithms that address deficiencies found in previous heuristics, (3) implement an object-oriented library of these algorithms in a robust, modular fashion without significant loss of efficiency, and (4) extend our algorithms and software to address both generalized and constrained problems. We stress that the major contribution is the algorithms and the implementation; the whole being greater than the sum of its parts. The initial motivation for implementing our algorithms in object-oriented software was to manage the inherent complexity. During our research came the realization that the object-oriented implementation enabled new possibilities for augmented algorithms that would not have been as natural to generalize from a procedural implementation. Some extensions are constructed from a family of related algorithmic components, thereby creating a poly-algorithm that can adapt its strategy to the properties of the specific problem instance dynamically. Other algorithms are tailored for special constraints by aggregating algorithmic components and having them collaboratively generate the global ordering. Our software laboratory, “Spindle,” implements state-of-the-art ordering algorithms for sparse matrices and graphs. We have used it to examine and augment the behavior of existing algorithms and test new ones. Its 40,000+ lines of C++ code includes a base library test drivers, sample applications, and interfaces to C, C++, Matlab, and PETSc. Spindle is freely available and can be built on a variety of UNIX platforms as well as WindowsNT

SCEE 2008 book of abstracts : the 7th International Conference on Scientific Computing in Electrical Engineering (SCEE 2008), September 28 – October 3, 2008, Helsinki University of Technology, Espoo, Finland

This report contains abstracts of presentations given at the SCEE 2008 conference.reviewe

Contribution to quality of user experience provision over wireless networks

The widespread expansion of wireless networks has brought new attractive possibilities to end users. In addition to the mobility capabilities provided by unwired devices, it is worth remarking the easy configuration process that a user has to follow to gain connectivity through a wireless network. Furthermore, the increasing bandwidth provided by the IEEE 802.11 family has made possible accessing to high-demanding services such as multimedia communications. Multimedia traffic has unique characteristics that make it greatly vulnerable against network impairments, such as packet losses, delay, or jitter. Voice over IP (VoIP) communications, video-conference, video-streaming, etc., are examples of these high-demanding services that need to meet very strict requirements in order to be served with acceptable levels of quality. Accomplishing these tough requirements will become extremely important during the next years, taking into account that consumer video traffic will be the predominant traffic in the Internet during the next years. In wired systems, these requirements are achieved by using Quality of Service (QoS) techniques, such as Differentiated Services (DiffServ), traffic engineering, etc. However, employing these methodologies in wireless networks is not that simple as many other factors impact on the quality of the provided service, e.g., fading, interferences, etc. Focusing on the IEEE 802.11g standard, which is the most extended technology for Wireless Local Area Networks (WLANs), it defines two different architecture schemes. On one hand, the infrastructure mode consists of a central point, which manages the network, assuming network controlling tasks such as IP assignment, routing, accessing security, etc. The rest of the nodes composing the network act as hosts, i.e., they send and receive traffic through the central point. On the other hand, the IEEE 802.11 ad-hoc configuration mode is less extended than the infrastructure one. Under this scheme, there is not a central point in the network, but all the nodes composing the network assume both host and router roles, which permits the quick deployment of a network without a pre-existent infrastructure. This type of networks, so called Mobile Ad-hoc NETworks (MANETs), presents interesting characteristics for situations when the fast deployment of a communication system is needed, e.g., tactics networks, disaster events, or temporary networks. The benefits provided by MANETs are varied, including high mobility possibilities provided to the nodes, network coverage extension, or network reliability avoiding single points of failure. The dynamic nature of these networks makes the nodes to react to topology changes as fast as possible. Moreover, as aforementioned, the transmission of multimedia traffic entails real-time constraints, necessary to provide these services with acceptable levels of quality. For those reasons, efficient routing protocols are needed, capable of providing enough reliability to the network and with the minimum impact to the quality of the service flowing through the nodes. Regarding quality measurements, the current trend is estimating what the end user actually perceives when consuming the service. This paradigm is called Quality of user Experience (QoE) and differs from the traditional Quality of Service (QoS) approach in the human perspective given to quality estimations. In order to measure the subjective opinion that a user has about a given service, different approaches can be taken. The most accurate methodology is performing subjective tests in which a panel of human testers rates the quality of the service under evaluation. This approach returns a quality score, so-called Mean Opinion Score (MOS), for the considered service in a scale 1 - 5. This methodology presents several drawbacks such as its high expenses and the impossibility of performing tests at real time. For those reasons, several mathematical models have been presented in order to provide an estimation of the QoE (MOS) reached by different multimedia services In this thesis, the focus is on evaluating and understanding the multimedia-content transmission-process in wireless networks from a QoE perspective. To this end, firstly, the QoE paradigm is explored aiming at understanding how to evaluate the quality of a given multimedia service. Then, the influence of the impairments introduced by the wireless transmission channel on the multimedia communications is analyzed. Besides, the functioning of different WLAN schemes in order to test their suitability to support highly demanding traffic such as the multimedia transmission is evaluated. Finally, as the main contribution of this thesis, new mechanisms or strategies to improve the quality of multimedia services distributed over IEEE 802.11 networks are presented. Concretely, the distribution of multimedia services over ad-hoc networks is deeply studied. Thus, a novel opportunistic routing protocol, so-called JOKER (auto-adJustable Opportunistic acK/timEr-based Routing) is presented. This proposal permits better support to multimedia services while reducing the energy consumption in comparison with the standard ad-hoc routing protocols.Universidad Politécnica de CartagenaPrograma Oficial de Doctorado en Tecnologías de la Información y Comunicacione