9 research outputs found
Global convergence and limit cycle behavior of weights of perceptron
In this paper, it is found that the weights of a perceptron are bounded for all initial weights if there exists a nonempty set of initial weights that the weights of the perceptron are bounded. Hence, the boundedness condition of the weights of the perceptron is independent of the initial weights. Also, a necessary and sufficient condition for the weights of the perceptron exhibiting a limit cycle behavior is derived. The range of the number of updates for the weights of the perceptron required to reach the limit cycle is estimated. Finally, it is suggested that the perceptron exhibiting the limit cycle behavior can be employed for solving a recognition problem when downsampled sets of bounded training feature vectors are linearly separable. Numerical computer simulation results show that the perceptron exhibiting the limit cycle behavior can achieve a better recognition performance compared to a multilayer perceptro
Incremental and stable training algorithm for wind turbine neural modeling
Training and topology design of artificial neuralnetworks are important issues with largeapplication. This paper deals with an improvedalgorithm for feed forward neural networks (FNN)聽straining. The association of an incrementalapproach and the Lyapunov stability theoryaccomplishes both good generalization and stabletraining process. The algorithm is tested on windturbine modeling. Compared to the incrementalapproach and to the Lyapunov stability basedmethod, the association of both strategies givesinteresting results
Adaptive Analytical Approach to Lean and Green Operations
Recent problems faced by industrial players commonly relates to global warming and depletion of resources. This situation highlights the importance of improvement solutions for industrial operations and environmental performances. Based on interviews and literature studies, manpower, machine, material, money and environment are known as the foundation resources to fulfil the facility's operation. The most critical and common challenge that is being faced by the industrialists is to perform continuous improvement effectively. The needs to develop a systematic framework to assist and guide the industrialist to achieve lean and green is growing rapidly. In this paper, a novel development of an adaptive analytic model for lean and green operation and processing is presented. The development of lean and green index will act as a benchmarking tool for the industrialist. This work uses the analytic hierarchy process to obtain experts opinion in determining the priority of the lean and green components and indicators. The application of backpropagation optimisation method will further enhance the lean and green model in guiding the industrialist for continuous improvement. An actual industry case study (combine heat and power plant) will be presented with the proposed lean and green model. The model is expected to enhance processing plant performance in a systematic lean and green manner
Adaptive Analytical Approach to Lean and Green Operations
Recent problems faced by industrial players commonly relates to global warming and depletion of resources. This situation highlights the importance of improvement solutions for industrial operations and environmental performances. Based on interviews and literature studies, manpower, machine, material, money and environment are known as the foundation resources to fulfil the facility's operation. The most critical and common challenge that is being faced by the industrialists is to perform continuous improvement effectively. The needs to develop a systematic framework to assist and guide the industrialist to achieve lean and green is growing rapidly. In this paper, a novel development of an adaptive analytic model for lean and green operation and processing is presented. The development of lean and green index will act as a benchmarking tool for the industrialist. This work uses the analytic hierarchy process to obtain experts opinion in determining the priority of the lean and green components and indicators. The application of backpropagation optimisation method will further enhance the lean and green model in guiding the industrialist for continuous improvement. An actual industry case study (combine heat and power plant) will be presented with the proposed lean and green model. The model is expected to enhance processing plant performance in a systematic lean and green manner
Models for time series prediction based on neural networks. Case study : GLP sales prediction from ANCAP.
A time series is a sequence of real values that can be considered as observations of a certain
system. In this work, we are interested in time series coming from dynamical systems. Such
systems can be sometimes described by a set of equations that model the underlying mechanism
from where the samples come. However, in several real systems, those equations are
unknown, and the only information available is a set of temporal measures, that constitute
a time series. On the other hand, by practical reasons it is usually required to have a prediction,
v.g. to know the (approximated) value of the series in a future instant t. The goal of
this thesis is to solve one of such real-world prediction problem: given historical data related
with the lique ed bottled propane gas sales, predict the future gas sales, as accurately as
possible. This time series prediction problem is addressed by means of neural networks,
using both (dynamic) reconstruction and prediction. The problem of to dynamically reconstruct
the original system consists in building a model that captures certain characteristics
of it in order to have a correspondence between the long-term behavior of the model and of
the system.
The networks design process is basically guided by three ingredients. The dimensionality
of the problem is explored by our rst ingredient, the Takens-Ma帽茅's theorem. By means
of this theorem, the optimal dimension of the (neural) network input can be investigated.
Our second ingredient is a strong theorem: neural networks with a single hidden layer are
universal approximators. As the third ingredient, we faced the search of the optimal size
of the hidden layer by means of genetic algorithms, used to suggest the number of hidden
neurons that maximizes a target tness function (related with prediction errors). These
algorithms are also used to nd the most in uential networks inputs in some cases. The
determination of the hidden layer size is a central (and hard) problem in the determination
of the network topology.
This thesis includes a state of the art of neural networks design for time series prediction, including
related topics such as dynamical systems, universal approximators, gradient-descent
searches and variations, as well as meta-heuristics. The survey of the related literature is
intended to be extensive, for both printed material and electronic format, in order to have a
landscape of the main aspects for the state of the art in time series prediction using neural
networks. The material found was sometimes extremely redundant (as in the case of the
back-propagation algorithm and its improvements) and scarce in others (memory structures
or estimation of the signal subspace dimension in the stochastic case). The surveyed literature
includes classical research works ([27], [50], [52]) as well as more recent ones ([79] , [16]
or [82]), which pretends to be another contribution of this thesis.
Special attention is given to the available software tools for neural networks design and time
series processing. After a review of the available software packages, the most promising
computational tools for both approaches are discussed. As a result, a whole framework
based on mature software tools was set and used. In order to work with such dynamical
systems, software intended speci cally for the analysis and processing of time series was
employed, and then chaotic series were part of our focus.
Since not all randomness is attributable to chaos, in order to characterize the dynamical
system generating the time series, an exploration of chaotic-stochastic systems is required,
as well as network models to predict a time series associated to one of them. Here we
pretend to show how the knowledge of the domain, something extensively treated in the
bibliography, can be someway sophisticated (such as the Lyapunov's spectrum for a series
or the embedding dimension). In order to model the dynamical system generated by the time series we used the state-space model, so the time series prediction was translated in the prediction of the next system
state. This state-space model, together with the delays method (delayed coordinates) have
practical importance for the development of this work, speci cally, the design of the input
layer in some networks (multi-layer perceptrons - MLPs) and other parameters (taps in the
TFLNs). Additionally, the rest of the network components where determined in many cases
through procedures traditionally used in neural networks : genetic algorithms.
The criteria of model (network) selection are discussed and a trade-o between performance
and network complexity is further explored, inspired in the Rissanen's minimum description
length and its estimation given by the chosen software. Regarding the employed network
models, the network topologies suggested from the literature as adequate for the prediction
are used (TLFNs and recurrent networks) together with MLPs (a classic of arti cial neural
networks) and networks committees. The e ectiveness of each method is con rmed for the
proposed prediction problem. Network committees, where the predictions are a naive convex
combination of predictions from individual networks, are also extensively used.
The need of criteria to compare the behaviors of the model and of the real system, in the long
run, for a dynamic stochastic systems, is presented and two alternatives are commented.
The obtained results proof the existence of a solution to the problem of learning of the
dependence Input ! Output . We also conjecture that the system is dynamic-stochastic
but not chaotic, because we only have a realization of the random process corresponding to
the sales. As a non-chaotic system, the mean of the predictions of the sales would improve
as the available data increase, although the probability of a prediction with a big error is
always non-null due to the randomness present. This solution is found in a constructive and
exhaustive way. The exhaustiveness can be deduced from the next ve statements:
the design of a neural network requires knowing the input and output dimension,the
number of the hidden layers and of the neurons in each of them.
the use of the Takens-Ma帽茅's theorem allows to derive the dimension of the input data
by theorems such as the Kolmogorov's and Cybenko's ones the use of multi-layer
perceptrons with only one hidden layer is justi ed so several of such models were
tested
the number of neurons in the hidden layer is determined many times heuristically
using genetic algorithms
a neuron in the output gives the desired prediction
As we said, two tasks are carried out: the development of a time series prediction model
and the analysis of a feasible model for the dynamic reconstruction of the system. With
the best predictive model, obtained by an ensemble of two networks, an acceptable average
error was obtained when the week to be predicted is not adjacent to the training set (7.04%
for the week 46/2011). We believe that these results are acceptable provided the quantity
of information available, and represent an additional validation that neural networks are
useful for time series prediction coming from dynamical systems, no matter whether they
are stochastic or not.
Finally, the results con rmed several already known facts (such as that adding noise to the
inputs and outputs of the training values can improve the results; that recurrent networks
trained with the back-propagation algorithm don't have the problem of vanishing gradients
in short periods and that the use of committees - which can be seen as a very basic of
distributed arti cial intelligence - allows to improve signi cantly the predictions).Una serie temporal es una secuencia de valores reales que pueden ser considerados como observaciones
de un cierto sistema. En este trabajo, estamos interesados en series temporales
provenientes de sistemas din谩micos. Tales sistemas pueden ser algunas veces descriptos por
un conjunto de ecuaciones que modelan el mecanismo subyacente que genera las muestras.
sin embargo, en muchos sistemas reales, esas ecuaciones son desconocidas, y la 煤nica informaci贸n disponible es un conjunto de medidas en el tiempo, que constituyen la serie temporal.
Por otra parte, por razones pr谩cticas es generalmente requerida una predicci贸n, es decir,
conocer el valor (aproximado) de la serie en un instante futuro t. La meta de esta tesis es
resolver un problema de predicci贸n del mundo real: dados los datos hist贸ricos relacionados
con las ventas de gas propano licuado, predecir las ventas futuras, tan aproximadamente
como sea posible. Este problema de predicci贸n de series temporales es abordado por medio
de redes neuronales, tanto para la reconstrucci贸n como para la predicci贸n. El problema de
reconstruir din谩micamente el sistema original consiste en construir un modelo que capture
ciertas caracter铆sticas de 茅l de forma de tener una correspondencia entre el comportamiento
a largo plazo del modelo y del sistema.
El proceso de dise帽o de las redes es guiado b谩sicamente por tres ingredientes. La dimensionalidad
del problema es explorada por nuestro primer ingrediente, el teorema de Takens-Ma帽茅.
Por medio de este teorema, la dimensi贸n 贸ptima de la entrada de la red neuronal puede ser
investigada. Nuestro segundo ingrediente es un teorema muy fuerte: las redes neuronales
con una sola capa oculta son un aproximador universal. Como tercer ingrediente, encaramos
la b煤squeda del tama帽o oculta de la capa oculta por medio de algoritmos gen茅ticos, usados
para sugerir el n煤mero de neuronas ocultas que maximizan una funci贸n objetivo (relacionada
con los errores de predicci贸n). Estos algoritmos se usan adem谩s para encontrar las entradas
a la red que influyen m谩s en la salida en algunos casos. La determinaci贸n del tama帽o de la
capa oculta es un problema central (y duro) en la determinaci贸n de la topolog铆a de la red.
Esta tesis incluye un estado del arte del dise帽o de redes neuronales para la predicci贸n de series
temporales, incluyendo t贸picos relacionados tales como sistemas din谩micos, aproximadores
universales, b煤squedas basadas en el gradiente y sus variaciones, as铆 como meta-heur铆sticas.
El relevamiento de la literatura relacionada busca ser extenso, para tanto el material impreso
como para el que esta en formato electr贸nico, de forma de tener un panorama de los
principales aspectos del estado del arte en la predicci贸n de series temporales usando redes
neuronales. El material hallado fue algunas veces extremadamente redundante (como en
el caso del algoritmo de retropropagaci贸n y sus mejoras) y escaso en otros (estructuras de
memoria o estimaci贸n de la dimensi贸n del sub-espacio de se帽al en el caso estoc谩stico). La
literatura consultada incluye trabajos de investigaci贸n cl谩sicos ( ([27], [50], [52])' as铆 como
de los m谩s reciente ([79] , [16] or [82]).
Se presta especial atenci贸n a las herramientas de software disponibles para el dise帽o de redes
neuronales y el procesamiento de series temporales. Luego de una revisi贸n de los paquetes
de software disponibles, las herramientas m谩s promisiorias para ambas tareas son discutidas.
Como resultado, un entorno de trabajo completo basado en herramientas de software maduras fue definido y usado. Para trabajar con los mencionados sistemas din谩micos, software
especializado en el an谩lisis y proceso de las series temporales fue empleado, y entonces
las series ca贸ticas fueron estudiadas.
Ya que no toda la aleatoriedad es atribuible al caos, para caracterizar al sistema din谩mico
que genera la serie temporal se requiere una exploraci贸n de los sistemas ca贸ticos-estoc谩sticos,
as铆 como de los modelos de red para predecir una serie temporal asociada a uno de ellos.
Aqu铆 se pretende mostrar c贸mo el conocimiento del dominio, algo extensamente tratado en
la literatura, puede ser de alguna manera sofisticado (tal como el espectro de Lyapunov de
la serie o la dimensi贸n del sub-espacio de se帽al).
Para modelar el sistema din谩mico generado por la serie temporal se usa el modelo de espacio
de estados, por lo que la predicci贸n de la serie temporal es traducida en la predicci贸n
del siguiente estado del sistema. Este modelo de espacio de estados, junto con el m茅todo
de los delays (coordenadas demoradas) tiene importancia pr谩ctica en el desarrollo de este
trabajo, espec铆ficamente, en el dise帽o de la capa de entrada en algunas redes (los perceptrones
multicapa) y otros par谩metros (los taps de las redes TLFN). Adicionalmente, el resto
de los componentes de la red con determinados en varios casos a trav茅s de procedimientos
tradicionalmente usados en las redes neuronales: los algoritmos gen茅ticos.
Los criterios para la selecci贸n de modelo (red) son discutidos y un balance entre performance
y complejidad de la red es explorado luego, inspirado en el minimum description length de
Rissanen y su estimaci贸n dada por el software elegido.
Con respecto a los modelos de red empleados, las topolog贸as de sugeridas en la literatura
como adecuadas para la predicci贸n son usadas (TLFNs y redes recurrentes) junto con perceptrones
multicapa (un cl谩sico de las redes neuronales) y comit茅s de redes. La efectividad
de cada m茅todo es confirmada por el problema de predicci贸n propuesto. Los comit茅s de
redes, donde las predicciones son una combinaci贸n convexa de las predicciones dadas por
las redes individuales, son tambi茅n usados extensamente.
La necesidad de criterios para comparar el comportamiento del modelo con el del sistema
real, a largo plazo, para un sistema din谩mico estoc谩stico, es presentada y dos alternativas
son comentadas.
Los resultados obtenidos prueban la existencia de una soluci贸n al problema del aprendizaje
de la dependencia Entrada - Salida . Conjeturamos adem谩s que el sistema generador de
serie de las ventas es din谩mico-estoc谩stico pero no ca贸tico, ya que s贸lo tenemos una realizaci贸n del proceso aleatorio correspondiente a las ventas. Al ser un sistema no ca贸tico, la media de las predicciones de las ventas deber铆a mejorar a medida que los datos disponibles
aumentan, aunque la probabilidad de una predicci贸n con un gran error es siempre no nula debido
a la aleatoriedad presente. Esta soluci贸n es encontrada en una forma constructiva
y exhaustiva. La exhaustividad puede deducirse de las siguiente cinco afirmaciones :
el dise帽o de una red neuronal requiere conocer la dimensi贸n de la entrada y de la
salida, el n煤mero de capas ocultas y las neuronas en cada una de ellas
el uso del teorema de takens-Ma帽茅 permite derivar la dimensi贸n de la entrada
por teoremas tales como los de Kolmogorov y Cybenko el uso de perceptrones con solo
una capa oculta es justificado, por lo que varios de tales modelos son probados
el n煤mero de neuronas en la capa oculta es determinada varias veces heur铆sticamente
a trav茅s de algoritmos gen茅ticos
una sola neurona de salida da la predicci贸n deseada. Como se dijo, dos tareas son llevadas a cabo: el desarrollo de un modelo para la predicci贸n de la serie temporal y el an谩lisis de un modelo factible para la reconstrucci贸n din谩mica del sistema. Con el mejor modelo predictivo, obtenido por el comit茅 de dos redes se logr贸 obtener un error aceptable en la predicci贸n de una semana no contigua al conjunto de
entrenamiento (7.04% para la semana 46/2011). Creemos que este es un resultado aceptable
dada la cantidad de informaci贸n disponible y representa una validaci贸n adicional de que las
redes neuronales son 煤tiles para la predicci贸n de series temporales provenientes de sistemas
din谩micos, sin importar si son estoc谩sticos o no.
Finalmente, los resultados experimentales confirmaron algunos hechos ya conocidos (tales
como que agregar ruido a los datos de entrada y de salida de los valores de entrenamiento
puede mejorar los resultados: que las redes recurrentes entrenadas con el algoritmo de
retropropagaci贸n no presentan el problema del gradiente evanescente en periodos cortos y
que el uso de de comit茅s - que puede ser visto como una forma muy b谩sica de inteligencia
artificial distribuida - permite mejorar significativamente las predicciones)
Global parameter identification and control of nonlinearly parameterized systems
Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2002.Includes bibliographical references (leaves 109-114).Nonlinearly parameterized (NLP) systems are ubiquitous in nature and many fields of science and engineering. Despite the wide and diverse range of applications, there exist relatively few results in control systems literature which exploit the structure of the nonlinear parameterization. A vast majority of presently applicable global control design approaches to systems with NLP, make use of either feedback-linearization, or assume linear parameterization, and ignore the specific structure of the nonlinear parameterization. While this type of approach may guarantee stability, it introduced three major drawbacks. First, they produce no additional information about the nonlinear parameters. Second, they may require large control authority and actuator bandwidth, which makes them unsuitable for some applications. Third, they may simply result in unacceptably poor performance. All of these inadequacies are amplified further when parametric uncertainties are present. What is necessary is a systematic adaptive approach to identification and control of such systems that explicitly accommodates the presence of nonlinear parameters that may not be known precisely. This thesis presents results in both adaptive identification and control of NLP systems. An adaptive controller is presented for NLP systems with a triangular structure. The presence of the triangular structure together with nonlinear parameterization makes standard methods such as back-stepping, and variable structure control inapplicable. A concept of bounding functions is combined with min-max adaptation strategies and recursive error formulation to result in a globally stabilizing controller.(cont.) A large class of nonlinear systems including cascaded LNL (linear-nonlinear-linear) systems are shown to be controllable using this approach. In the context of parameter identification, results are derived for two classes of NLP systems. The first concerns systems with convex/concave parameterization, where min-max algorithms are essential for global stability. Stronger conditions of persistent excitation are shown to be necessary to overcome the presence of multiple equilibrium points which are introduced due to the stabilization aspects of the min-max algorithms. These conditions imply that the min-max estimator must periodically employ the local gradient information in order to guarantee parameter convergence. The second class of NLP systems considered in this concerns monotonically parameterized systems, of which neural networks are a specific example. It is shown that a simple algorithm based on local gradient information suffices for parameter identification. Conditions on the external input under which the parameter estimates converge to the desired set starting from arbitrary values are derived. The proof makes direct use of the monotonicity in the parameters, which in turn allows local gradients to be self-similar and therefore introduces a desirable invariance property. By suitably exploiting this invariance property and defining a sequence of distance metrics, global convergence is proved. Such a proof of global convergence is in contrast to most other existing results in the area of nonlinear parameterization, in general, and neural networks in particular.by Aleksandar M. Koji脛.Ph.D
Nuevo enfoque en el dise帽o y entrenamiento de redes neuronales para la clasificaci贸n
Tesis (Doctor en Ingenier铆a con Especialidad en Ingenier铆a de Sistemas) UANL, 2001.UANLhttp://www.uanl.mx
Optimal convergence of on-line backpropagation
Many researchers are quite skeptical about the actual behavior of neural network learning algorithms like backpropagation. One of the major problems is with the lack of clear theoretical results on optimal convergence, particularly for pattern mode algorithms. In this paper, we prove the companion of Rosenblatt's PC (perceptron convergence) theorem for feedforward networks (1960), stating that pattern mode backpropagation converges to an optimal solution for linearly separable patterns