Time series prediction and channel equalizer using artificial neural networks with VLSI implementation

The architecture and training procedure of a novel recurrent neural network (RNN), referred to as the multifeedbacklayer neural network (MFLNN), is described in this paper. The main difference of the proposed network compared to the available RNNs is that the temporal relations are provided by means of neurons arranged in three feedback layers, not by simple feedback elements, in order to enrich the representation capabilities of the recurrent networks. The feedback layers provide local and global recurrences via nonlinear processing elements. In these feedback layers, weighted sums of the delayed outputs of the hidden and of the output layers are passed through certain activation functions and applied to the feedforward neurons via adjustable weights. Both online and offline training procedures based on the backpropagation through time (BPTT) algorithm are developed. The adjoint model of the MFLNN is built to compute the derivatives with respect to the MFLNN weights which are then used in the training procedures. The Levenberg–Marquardt (LM) method with a trust region approach is used to update the MFLNN weights. The performance of the MFLNN is demonstrated by applying to several illustrative temporal problems including chaotic time series prediction and nonlinear dynamic system identification, and it performed better than several networks available in the literature

Center for Aeronautics and Space Information Sciences

This report summarizes the research done during 1991/92 under the Center for Aeronautics and Space Information Science (CASIS) program. The topics covered are computer architecture, networking, and neural nets

Adaptive equalisation for fading digital communication channels

This thesis considers the design of new adaptive equalisers for fading digital communication channels. The role of equalisation is discussed in the context of the functions of a digital radio communication system and both conventional and more recent novel equaliser designs are described. The application of recurrent neural networks to the problem of equalisation is developed from a theoretical study of a single node structure to the design of multinode structures. These neural networks are shown to cancel intersymbol interference in a manner mimicking conventional techniques and simulations demonstrate their sensitivity to symbol estimation errors. In addition the error mechanisms of conventional maximum likelihood equalisers operating on rapidly time-varying channels are investigated and highlight the problems of channel estimation using delayed and often incorrect symbol estimates. The relative sensitivity of Bayesian equalisation techniques to errors in the channel estimate is studied and demonstrates that the structure's equalisation capability is also susceptible to such errors. Applications of multiple channel estimator methods are developed, leading to reduced complexity structures which trade performance for a smaller computational load. These novel structures are shown to provide an improvement over the conventional techniques, especially for rapidly time-varying channels, by reducing the time delay in the channel estimation process. Finally, the use of confidence measures of the equaliser's symbol estimates in order to improve channel estimation is studied and isolates the critical areas in the development of the technique — the production of reliable confidence measures by the equalisers and the statistics of symbol estimation error bursts

Dynamically reconfigurable bio-inspired hardware

During the last several years, reconfigurable computing devices have experienced an impressive development in their resource availability, speed, and configurability. Currently, commercial FPGAs offer the possibility of self-reconfiguring by partially modifying their configuration bitstream, providing high architectural flexibility, while guaranteeing high performance. These configurability features have received special interest from computer architects: one can find several reconfigurable coprocessor architectures for cryptographic algorithms, image processing, automotive applications, and different general purpose functions. On the other hand we have bio-inspired hardware, a large research field taking inspiration from living beings in order to design hardware systems, which includes diverse topics: evolvable hardware, neural hardware, cellular automata, and fuzzy hardware, among others. Living beings are well known for their high adaptability to environmental changes, featuring very flexible adaptations at several levels. Bio-inspired hardware systems require such flexibility to be provided by the hardware platform on which the system is implemented. In general, bio-inspired hardware has been implemented on both custom and commercial hardware platforms. These custom platforms are specifically designed for supporting bio-inspired hardware systems, typically featuring special cellular architectures and enhanced reconfigurability capabilities; an example is their partial and dynamic reconfigurability. These aspects are very well appreciated for providing the performance and the high architectural flexibility required by bio-inspired systems. However, the availability and the very high costs of such custom devices make them only accessible to a very few research groups. Even though some commercial FPGAs provide enhanced reconfigurability features such as partial and dynamic reconfiguration, their utilization is still in its early stages and they are not well supported by FPGA vendors, thus making their use difficult to include in existing bio-inspired systems. In this thesis, I present a set of architectures, techniques, and methodologies for benefiting from the configurability advantages of current commercial FPGAs in the design of bio-inspired hardware systems. Among the presented architectures there are neural networks, spiking neuron models, fuzzy systems, cellular automata and random boolean networks. For these architectures, I propose several adaptation techniques for parametric and topological adaptation, such as hebbian learning, evolutionary and co-evolutionary algorithms, and particle swarm optimization. Finally, as case study I consider the implementation of bio-inspired hardware systems in two platforms: YaMoR (Yet another Modular Robot) and ROPES (Reconfigurable Object for Pervasive Systems); the development of both platforms having been co-supervised in the framework of this thesis

Higher order neural networks for financial time series prediction

Neural networks have been shown to be a promising tool for forecasting financial times series. Numerous research and applications of neural networks in business have proven their advantage in relation to classical methods that do not include artificial intelligence. What makes this particular use of neural networks so attractive to financial analysts and traders is the fact that governments and companies benefit from it to make decisions on investment and trading. However, when the number of inputs to the model and the number of training examples becomes extremely large, the training procedure for ordinary neural network architectures becomes tremendously slow and unduly tedious. To overcome such time-consuming operations, this research work focuses on using various Higher Order Neural Networks (HONNs) which have a single layer of learnable weights, therefore reducing the networks' complexity. In order to predict the upcoming trends of univariate financial time series signals, three HONNs models; the Pi-Sigma Neural Network, the Functional Link Neural Network, and the Ridge Polynomial Neural Network were used, as well as the Multilayer Perceptron. Furthermore, a novel neural network architecture which comprises of a feedback connection in addition to the feedforward Ridge Polynomial Neural Network was constructed. The proposed network combines the properties of both higher order and recurrent neural networks, and is called Dynamic Ridge Polynomial Neural Network (DRPNN). Extensive simulations covering ten financial time series were performed. The forecasting performance of various feedforward HONNs models, the Multilayer Perceptron and the novel DRPNN was compared. Simulation results indicate that HONNs, particularly the DRPNN in most cases demonstrated advantages in capturing chaotic movement in the financial signals with an improvement in the profit return over other network models. The relative superiority of DRPNN to other networks is not just its ability to attain high profit return, but rather to model the training set with fast learning and convergence. The network offers fast training and shows considerable promise as a forecasting tool. It is concluded that DRPNN do have the capability to forecast the financial markets, and individual investor could benefit from the use of this forecasting

Exploring Liquid Computing in a Hardware Adaptation : Construction and Operation of a Neural Network Experiment

Future increases in computing power strongly rely on miniaturization, large scale integration, and parallelization. Yet, approaching the nanometer realm poses new challenges in terms of device reliability, power dissipation, and connectivity - issues that have been of lesser concern in today's prevailing microprocessor implementations. It is therefore necessary to pursue the research on alternative computing architectures and strategies that can make use of large numbers of unreliable devices and only have a moderate power consumption. This thesis describes the construction of an experiment dedicated to exploring silicon adaptations of artificial neural network paradigms for their general applicability, power efficiency, and fault-tolerance. The presented setup comprises peripheral electronics, programmable logic, and software to accommodate a mixed-signal CMOS microchip implementing a flexible perceptron with 256 McCulloch-Pitts neurons. This neural network experiment is used to explore a recent strategy that allows to access the power of recurrent network topologies. While it has been conjectured that this liquid computing is suited for hardware implementations, this first time adaptation to a CMOS neural network affirms this claim. Not only feasibility but also tolerance to substrate variations and robustness to faults during operation are demonstrated

Learning for Network Applications and Control

The emergence of new Internet applications and technologies have resulted in an increased complexity as well as a need for lower latency, higher bandwidth, and increased reliability. This ultimately results in an increased complexity of network operation and management. Manual management is not sufficient to meet these new requirements. There is a need for data driven techniques to advance from manual management to autonomous management of network systems. One such technique, Machine Learning (ML), can use data to create models from hidden patterns in the data and make autonomous modifications. This approach has shown significant improvements in other domains (e.g., image recognition and natural language processing). The use of ML, along with advances in programmable control of Software- Defined Networks (SDNs), will alleviate manual network intervention and ultimately aid in autonomous network operations. However, realizing a data driven system that can not only understand what is happening in the network but also operate autonomously requires advances in the networking domain, as well as in ML algorithms. In this thesis, we focus on developing ML-based network architectures and data driven net- working algorithms whose objective is to improve the performance and management of future networks and network applications. We focus on problems spanning across the network protocol stack from the application layer to the physical layer. We design algorithms and architectures that are motivated by measurements and observations in real world or experimental testbeds. In Part I we focus on the challenge of monitoring and estimating user video quality of experience (QoE) of encrypted video traffic for network operators. We develop a system for REal-time QUality of experience metric detection for Encrypted Traffic, Requet. Requet uses a detection algorithm to identify video and audio chunks from the IP headers of encrypted traffic. Features extracted from the chunk statistics are used as input to a random forest ML model to predict QoE metrics. We evaluate Requet on a YouTube dataset we collected, consisting of diverse video assets delivered over various WiFi and LTE network conditions. We then extend Requet, and present a study on YouTube TV live streaming traffic behavior over WiFi and cellular networks covering a 9-month period. We observed pipelined chunk requests, a reduced buffer capacity, and a more stable chunk duration across various video resolutions compared to prior studies of on-demand streaming services. We develop a YouTube TV analysis tool using chunks statistics detected from the extracted data as input to a ML model to infer user QoE metrics. In Part II we consider allocating end-to-end resources in cellular networks. Future cellular networks will utilize SDN and Network Function Virtualization (NFV) to offer increased flexibility for network infrastructure operators to utilize network resources. Combining these technologies with real-time network load prediction will enable efficient use of network resources. Specifically, we leverage a type of recurrent neural network, Long Short-Term Memory (LSTM) neural networks, for (i) service specific traffic load prediction for network slicing, and (ii) Baseband Unit (BBU) pool traffic load prediction in a 5G cloud Radio Access Network (RAN). We show that leveraging a system with better accuracy to predict service requirements results in a reduction of operation costs. We focus on addressing the optical physical layer in Part III. Greater network flexibility through SDN and the growth of high bandwidth services are motivating faster service provisioning and capacity management in the optical layer. These functionalities require increased capacity along with rapid reconfiguration of network resources. Recent advances in optical hardware can enable a dramatic reduction in wavelength provisioning times in optical circuit switched networks. To support such operations, it is imperative to reconfigure the network without causing a drop in service quality to existing users. Therefore, we present a ML system that uses feedforward neural networks to predict the dynamic response of an optically circuit-switched 90-channel multi-hop Reconfigurable Optical Add-Drop Multiplexer (ROADM) network. We show that the trained deep neural network can recommend wavelength assignments for wavelength switching with minimal power excursions. We extend the performance of the ML system by implementing and testing a Hybrid Machine Learning (HML) model, which combines an analytical model with a neural network machine learning model to achieve higher prediction accuracy. In Part IV, we use a data-driven approach to address the challenge of wireless content delivery in crowded areas. We present the Adaptive Multicast Services (AMuSe) system, whose objective is to enable scalable and adaptive WiFi multicast. Specifically, we develop an algorithm for dynamic selection of a subset of the multicast receivers as feedback nodes. Further, we describe the Multicast Dynamic Rate Adaptation (MuDRA) algorithm that utilizes AMuSe’s feedback to optimally tune the physical layer multicast rate. Our experimental evaluation of MuDRA on the ORBIT testbed shows that MuDRA outperforms other schemes and supports high throughput multicast flows to hundreds of nodes while meeting quality requirements. We leverage the lessons learned from AMuSe for WiFi and use order statistics to address the performance issues with LTE evolved Multimedia Broadcast/Multicast Service (eMBMS). We present the Dynamic Monitoring (DyMo) system which provides low-overhead and real-time feedback about eMBMS performance to be used for network optimization. We focus on the Quality of Service (QoS) Evaluation module and develop a Two-step estimation algorithm which can efficiently identify the SNR Threshold as a one time estimation. DyMo significantly outperforms alternative schemes based on the Order-Statistics estimation method which relies on random or periodic sampling

Role of deep learning techniques in non-invasive diagnosis of human diseases.

Machine learning, a sub-discipline in the domain of artificial intelligence, concentrates on algorithms able to learn and/or adapt their structure (e.g., parameters) based on a set of observed data. The adaptation is performed by optimizing over a cost function. Machine learning obtained a great attention in the biomedical community because it offers a promise for improving sensitivity and/or specificity of detection and diagnosis of diseases. It also can increase objectivity of the decision making, decrease the time and effort on health care professionals during the process of disease detection and diagnosis. The potential impact of machine learning is greater than ever due to the increase in medical data being acquired, the presence of novel modalities being developed and the complexity of medical data. In all of these scenarios, machine learning can come up with new tools for interpreting the complex datasets that confront clinicians. Much of the excitement for the application of machine learning to biomedical research comes from the development of deep learning which is modeled after computation in the brain. Deep learning can help in attaining insights that would be impossible to obtain through manual analysis. Deep learning algorithms and in particular convolutional neural networks are different from traditional machine learning approaches. Deep learning algorithms are known by their ability to learn complex representations to enhance pattern recognition from raw data. On the other hand, traditional machine learning requires human engineering and domain expertise to design feature extractors and structure data. With increasing demands upon current radiologists, there are growing needs for automating the diagnosis. This is a concern that deep learning is able to address. In this dissertation, we present four different successful applications of deep learning for diseases diagnosis. All the work presented in the dissertation utilizes medical images. In the first application, we introduce a deep-learning based computer-aided diagnostic system for the early detection of acute renal transplant rejection. The system is based on the fusion of both imaging markers (apparent diffusion coefficients derived from diffusion-weighted magnetic resonance imaging) and clinical biomarkers (creatinine clearance and serum plasma creatinine). The fused data is then used as an input to train and test a convolutional neural network based classifier. The proposed system is tested on scans collected from 56 subjects from geographically diverse populations and different scanner types/image collection protocols. The overall accuracy of the proposed system is 92.9% with 93.3% sensitivity and 92.3% specificity in distinguishing non-rejected kidney transplants from rejected ones. In the second application, we propose a novel deep learning approach for the automated segmentation and quantification of the LV from cardiac cine MR images. We aimed at achieving lower errors for the estimated heart parameters compared to the previous studies by proposing a novel deep learning segmentation method. Using fully convolutional neural networks, we proposed novel methods for the extraction of a region of interest that contains the left ventricle, and the segmentation of the left ventricle. Following myocardial segmentation, functional and mass parameters of the left ventricle are estimated. Automated Cardiac Diagnosis Challenge dataset was used to validate our framework, which gave better segmentation, accurate estimation of cardiac parameters, and produced less error compared to other methods applied on the same dataset. Furthermore, we showed that our segmentation approach generalizes well across different datasets by testing its performance on a locally acquired dataset. In the third application, we propose a novel deep learning approach for automated quantification of strain from cardiac cine MR images of mice. For strain analysis, we developed a Laplace-based approach to track the LV wall points by solving the Laplace equation between the LV contours of each two successive image frames over the cardiac cycle. Following tracking, the strain estimation is performed using the Lagrangian-based approach. This new automated system for strain analysis was validated by comparing the outcome of these analysis with the tagged MR images from the same mice. There were no significant differences between the strain data obtained from our algorithm using cine compared to tagged MR imaging. In the fourth application, we demonstrate how a deep learning approach can be utilized for the automated classification of kidney histopathological images. Our approach can classify four classes: the fat, the parenchyma, the clear cell renal cell carcinoma, and the unusual cancer which has been discovered recently, called clear cell papillary renal cell carcinoma. Our framework consists of three convolutional neural networks and the whole-slide kidney images were divided into patches with three different sizes to be inputted to the networks. Our approach can provide patch-wise and pixel-wise classification. Our approach classified the four classes accurately and surpassed other state-of-the-art methods such as ResNet (pixel accuracy: 0.89 Resnet18, 0.93 proposed). In conclusion, the results of our proposed systems demonstrate the potential of deep learning for the efficient, reproducible, fast, and affordable disease diagnosis