XCS Algorithms for a Linear Combination of Discounted and Undiscounted Reward Markovian Decision Processes

RÉSUMÉ : Plusieurs études ont montré que combiner certains prédicteurs ensemble peut améliorer la justesse de la prédiction dans certains domaines comme la psychologie, les statistiques ou les sciences du management. Toutefois, aucune de ces études n'ont testé la combinaison de techniques d'apprentissage par renforcement. Notre étude vise à développer un algorithme basé sur deux algorithmes qui sont des formes approximatives d'apprentissage par renforcement répétés dans XCS. Cet algorithme, MIXCS, est une combinaison des techniques de Q-learning et de R-learning pour calculer la combinaison linéaire du payoff résultant des actions de l'agent, et aussi la correspondance entre la prédiction au niveau du système et la valeur réelle des actions de l'agent. MIXCS fait une prévision du payoff espéré pour chacune des actions disponibles pour l'agent. Nous avons testé MIXCS dans deux environnements à deux dimensions, Environment1 et Environment2, qui reproduisent les actions possibles dans un marché financier (acheter, vendre, ne rien faire) pour évaluer les performances d'un agent qui veut obtenir un profit espéré. Nous avons calculé le payoff optimal moyen dans nos deux environnements et avons comparé avec les résultats obtenus par MIXCS. Nous avons obtenu deux résultats. En premier, les résultats de MIXCS sont semblables au payoff optimal moyen pour Environments1, mais pas pour Environment2. Deuxièmement, l'agent obtient le payoff optimal moyen quand il prend l'action "vendre" dans les deux environnements.----------ABSTRACT : Many studies have shown that combining individual predictors improved the accuracy of predictions in different domains such as psychology, statistics and management sciences. However, these studies have not tested the combination of reinforcement learning techniques. This study aims to develop an algorithm based on two iterative approximate forms of reinforcement learning algorithm in XCS. This algorithm, named MIXCS, is a combination of Q-learning and R-learning techniques to compute the linear combination payoff and the correspondence between the system prediction and the action value. As such, MIXCS predicts the payoff to be expected for each possible action. We test MIXCS in two two-dimensional grids called Environment1 and Environment2, which represent financial markets actions of buying, selling and holding to evaluate the performance of an agent as a trader to gain the desired profit. We calculate the optimum average payoff to predict the value of the next movement in both Environment1 and Environment2 and compare the results with those obtained with MIXCS. The results show that the performance of MIXCS is close to optimum average reward in Environment1, but not in Environment2. Also, the agent reaches the maximum reward by taking selling actions in both Environments

Synthesis and Preformulation Studies of KTTKS and PAL-KTTKS as Anti-Wrinkle Peptides

Introduction: Skin aging is a complicated process which is one of the major issues in the field of dermatology and cosmetic products. Peptides are one of the novel ingredients included in the anti-aging formulations. KTTKS (Lys-Thr-Thr-Lys-Ser) and its derivative, PAL-KTTKS (Palmitic acid-KTTKS), have attracted a lot of attention in arresting or delaying skin aging, but unfortunately, there are almost no preformulation studies available about them. Methods and Results:Both peptides were synthesized by solid phase peptide synthesis and identified by Mass spectroscopy technique. UV absorption ability, percentage of crystallinity, melting point, decomposition temperature and thermal behavior of both peptides were analyzed by UV spectroscopy, XRD, TGA and DSC techniques respectively. Partition coefficient was also determined by ACD/chemsketch software. In addition stability studies for the aqueous solution of KTTKS were performed at 32 and 37 ℃.  The results of UV spectroscopy showsthe wavelength of maximum absorbance of both peptides is in the vacuum UV range. Based on the results of melting point and TGA apparatuses, KTTKS and PAL- KTTKS decompose at about 154 ℃ and 112 ℃ respectively and there is no melting point for them before decomposition. The results of DSC thermogramsindicate an endothermic peak at the temperature below 60 ℃ for both peptides which is probably due to intrinsic structural rearrangement or evaporation of volatile solvents. Crystallinity percentage for KTTKS and PAL-KTTKS are 62% and 32% respectively. cLogp of KTTKS is -3.27 and cLogp of PAL-KTTKS is 3.32. Conclusions: The results of this investigation can be employed for the formulation of these peptides for TTD

Quantitative assessment of wound healing using high-frequency ultrasound image analysis

Purpose: We aimed to develop a method for quantitative assessment of wound healing in ulcerated diabetic feet. Methods: High‐frequency ultrasound (HFU) images of 30 wounds were acquired in a controlled environment on post‐debridement days 7, 14, 21, and 28. Meaningful features portraying changes in structure and intensity of echoes during healing were extracted from the images, their relevance and discriminatory power being verified by analysis of variance. Relative analysis of tissue healing was conducted by developing a features‐based healing function, optimised using the pattern‐search method. Its performance was investigated through leave‐one‐out cross‐validation technique and reconfirmed using principal component analysis. Results: The constructed healing function could depict tissue changes during healing with 87.8% accuracy. The first principal component derived from the extracted features demonstrated similar pattern to the constructed healing function, accounting for 86.3% of the data variance. Conclusion: The developed wound analysis technique could be a viable tool in quantitative assessment of diabetic foot ulcers during healing

Information Loss Due to the Data Reduction of Sample Data from Discrete Distributions

In this paper, we study the information lost when a real-valued statistic is used to reduce or summarize sample data from a discrete random variable with a one-dimensional parameter. We compare the probability that a random sample gives a particular data set to the probability of the statistic’s value for this data set. We focus on sufficient statistics for the parameter of interest and develop a general formula independent of the parameter for the Shannon information lost when a data sample is reduced to such a summary statistic. We also develop a measure of entropy for this lost information that depends only on the real-valued statistic but neither the parameter nor the data. Our approach would also work for non-sufficient statistics, but the lost information and associated entropy would involve the parameter. The method is applied to three well-known discrete distributions to illustrate its implementation

Automated Two-Step Seamline Detection for Generating Large-Scale Orthophoto Mosaics from Drone Images

To generate an orthophoto mosaic from a collection of aerial images, the original images are first orthorectified individually using a Digital Surface Model (DSM). Then, they are stitched together along some determined seamlines to form the orthophoto mosaic. Determining appropriate seamlines is a critical process, as it affects the visual and geometric quality of the results. The stitching process can usually be done in frame-to-frame or multi-frame modes. Although the latter is more efficient, both still involve a lot of pre-processing, such as creating individual orthophotos, image registration, and overlap extraction. This paper presents a novel coarse-to-fine approach that directly determines the seamline network without such pre-processing. Our method has been specifically applied for UAV photogrammetry projects where, due to the large number of images and the corresponding overlaps, the orthophoto mosaic generation can be very challenging and time-consuming. We established the seamlines simultaneously for all the images through a two-step process. First, a DSM was generated, and a low-resolution grid was overlayed. Then, for each grid point, an optimal image was selected. Then, the grid cells are grouped into polygons based on their corresponding optimal image. Boundaries of these polygons established our seamline network. Thereafter, to generate the orthophoto mosaic, we overlayed a higher/full resolution grid on the top of the DSM, the optimal image of each point of which was quickly identified via our low-resolution polygons. In this approach, not only seamlines were automatically generated, but also were the need for the creation, registration, and overlap extraction of individual orthophotos. Our method was systematically compared with a conventional frame-to-frame (CF) technique from different aspects, including the number of double-mapped areas, discontinuities across the seamlines network, and the amount of processing time. The outcomes revealed a 46% decrease in orthophoto generation time and a notable reduction in the number of double-mapped areas, sawtooth effects, and object discontinuities within the constructed orthophoto mosaic

Optimization of producing oil and meal from canola seeds using microwave − pulsed electric field pretreatment

In this study, optimization of the extraction of canola seeds oil was investigated using microwave-pulsed electric field seeds pretreatment (MW-PEF) with different MW times (0 to 200 s) and PEF intensities (0 to 5 kV/cm). The seeds oil was then extracted using screw press with different speeds (11 to 57 rpm). Oil extraction efficiency, refractive index, peroxide and phenolic compounds of oil and meal protein were measured. Tocopherols content of the best sample was also measured. The results showed that the peroxide and phenolic compounds increased at higher time, intensity and speed. An increase in the MW time and PEF intensity at first led to an increase in the oil extraction efficiency and meal protein but then both parameters decreased. The efficiency of oil extraction and protein decreased at higher speeds. The refractive index of all samples was 1.475. Gamma tocopherol was predominate one in canola oil and applying the pretreatment led to an increase in the number of total tocopherols. Treating at 1.28 kV/cm for 140.5 s and 28.71 rpm was chosen as the optimum condition with high desirability (0.744)