Nature-Inspired Topology Optimization of Recurrent Neural Networks

Hand-crafting effective and efficient structures for recurrent neural networks (RNNs) is a difficult, expensive, and time-consuming process. To address this challenge, this work presents three nature-inspired (NI) algorithms for neural architecture search (NAS), introducing the subfield of nature-inspired neural architecture search (NI-NAS). These algorithms, based on ant colony optimization (ACO), progress from memory cell structure optimization, to bounded discrete-space architecture optimization, and finally to unbounded continuous-space architecture optimization. These methods were applied to real-world data sets representing challenging engineering problems, such as data from a coal-fired power plant, wind-turbine power generators, and aircraft flight data recorder (FDR) data. Initial work utilized ACO to select optimal connections inside recurrent long short-term memory (LSTM) cell structures. Viewing each LSTM cell as a graph, ants would choose potential input and output connections based on the pheromones previously laid down over those connections as done in a standard ACO search. However, this approach did not optimize the overall network of the RNN, particularly its synaptic parameters. I addressed this issue by introducing the Ant-based Neural Topology Search (ANTS) algorithm to directly optimize the entire RNN topology. ANTS utilizes a discrete-space superstructure representing a completely connected RNN where each node is connected to every other node, forming an extremely dense mesh of edges and recurrent edges. ANTS can select from a library of modern RNN memory cells. ACO agents (ants), in this thesis, build RNNs from the superstructure determined by pheromones laid out on the superstructure\u27s connections. Backpropagation is then used to train the generated RNNs in an asynchronous parallel computing design to accelerate the optimization process. The pheromone update depends on the evaluation of the tested RNN against a population of best performing RNNs. Several variations of the core algorithm was investigated to test several designed heuristics for ANTS and evaluate their efficacy in the formation of sparser synaptic connectivity patterns. This was done primarily by formulating different functions that drive the underlying pheromone simulation process as well as by introducing ant agents with 3 specialized roles (inspired by real-world ants) to construct the RNN structure. This characterization of the agents enables ants to focus on specific structure building roles. ``Communal intelligence\u27\u27 was also incorporated, where the best set of weights was across locally-trained RNN candidates for weight initialization, reducing the number of backpropagation epochs required to train each candidate RNN and speeding up the overall search process. However, the growth of the superstructure increased by an order of magnitude, as more input and deeper structures are utilized, proving to be one limitation of the proposed procedure. The limitation of ANTS motivated the development of the continuous ANTS algorithm (CANTS), which works with a continuous search space for any fixed network topology. In this process, ants moving within a (temporally-arranged) set of continuous/real-valued planes based on proximity and density of pheromone placements. The motion of the ants over these continuous planes, in a sense, more closely mimicks how actual ants move in the real world. Ants traverse a 3-dimensional space from the inputs to the outputs and across time lags. This continuous search space frees the ant agents from the limitations imposed by ANTS\u27 discrete massively connected superstructure, making the structural options unbounded when mapping the movements of ants through the 3D continuous space to a neural architecture graph. In addition, CANTS has fewer hyperparameters to tune than ANTS, which had five potential heuristic components that each had their own unique set of hyperparameters, as well as requiring the user to define the maximum recurrent depth, number of layers and nodes within each layer. CANTS only requires specifying the number ants and their pheromone sensing radius. The three applied strategies yielded three important successes. Applying ACO on optimizing LSTMs yielded a 1.34\% performance enhancement and more than 55% sparser structures (which is useful for speeding up inference). ANTS outperformed the NAS benchmark, NEAT, and the NAS state-of-the-art algorithm, EXAMM. CANTS showed competitive results to EXAMM and competed with ANTS while offering sparser structures, offering a promising path forward for optimizing (temporal) neural models with nature-inspired metaheuristics based the metaphor of ants

Chebyshev Collocation Method for Parabolic Partial Integrodifferential Equations

An efficient technique for solving parabolic partial integrodifferential equation is presented. This technique is based on Chebyshev polynomials and finite difference method. A priori error estimate for the proposed technique is deduced. Some examples are presented to illustrate the validity and efficiency of the presented method

Protein Phosphatase 2A Regulates Xanthine Oxidase-derived ROS Production in Macrophages and Influx of Inflammatory Monocytes in a Murine Gout Model

Background: Gout is a common arthritis, due to deposition of monosodium urate (MSU) crystals which results in IL-1β secretion by tissue-resident macrophages. Xanthine oxidase (XO) catalyzes uric acid (UA) production and in the process, reactive oxygen species (ROS) are generated which contributes to NLRP3 inflammasome activation. Protein phosphatase 2A (PP2A) may be involved in regulating inflammatory pathways in macrophages. The objective of this study was to investigate whether PP2A regulates gout inflammation, mediated by XO activity modulation. We studied UA and ROS generations in MSU stimulated murine bone marrow derived macrophages (BMDMs) in response to fingolimod phosphate, a PP2A activator, and compared its anti-inflammatory efficacy to that of an XO inhibitor, febuxostat. Methods: BMDMs were stimulated with MSU, GM-CSF/IL-1β or nigericin ± fingolimod (2.5 μM) or febuxostat (200 μM) and UA levels, ROS, XO, and PP2A activities, Xdh (XO) expression and secreted IL-1β levels were determined. PP2A activity and IL-1β in MSU stimulated BMDMs ± N-acetylcysteine (NAC) (10 μM) ± okadaic acid (a PP2A inhibitor) were also determined. M1 polarization of BMDMs in response to MSU ± fingolimod treatment was assessed by a combination of iNOS expression and multiplex cytokine assay. The in vivo efficacy of fingolimod was assessed in a murine peritoneal model of acute gout where peritoneal lavages were studied for pro-inflammatory classical monocytes (CMs), anti-inflammatory nonclassical monocytes (NCMs) and neutrophils by flow cytometry and IL-1β by ELISA. Results: Fingolimod reduced intracellular and secreted UA levels (p \u3c 0.05), Xdh expression (p \u3c 0.001), XO activity (p \u3c 0.001), ROS generation (p \u3c 0.0001) and IL-1β secretion (p \u3c 0.0001), whereas febuxostat enhanced PP2A activity (p \u3c 0.05). NAC treatment enhanced PP2A activity and reduced XO activity and PP2A restoration mediated NAC’s efficacy as co-treatment with okadaic acid increased IL-1β secretion (p \u3c 0.05). Nigericin activated caspase-1 and reduced PP2A activity (p \u3c 0.001) and fingolimod reduced caspase-1 activity in BMDMs (p \u3c 0.001). Fingolimod reduced iNOS expression (p \u3c 0.0001) and secretion of IL-6 and TNF-α (p \u3c 0.05). Fingolimod reduced CMs (p \u3c 0.0001), neutrophil (p \u3c 0.001) and IL-1β (p \u3c 0.05) lavage levels while increasing NCMs (p \u3c 0.001). Conclusion: Macrophage PP2A is inactivated in acute gout by ROS and a PP2A activator exhibited a broad anti-inflammatory effect in acute gout in vitro and in vivo

Iterative Solution of Fractional Diffusion Equation Modelling Anomalous Diffusion

In this article, we study the fractional diffusion equation with spatial Riesz fractional derivative. The continuation of the solution of this fractional equation to the solution of the corresponding integer order equation is proved. The series solution is obtained based on properties of Riesz fractional derivative operator and utilizing the optimal homotopy analysis method (OHAM). Numerical simulations are presented to validate the method and to show the effect of changing the fractional derivative parameter on the solution behavior

The role of osteopontin in children with systemic inflammatory response syndrome and sepsis

Introduction: Sepsis is a leading cause of morbidity and mortality in critically ill children despite the use of modern antibiotics and resuscitation therapies. Sepsis must be distinguished from non-infection systemic inflammatory response syndrome (SIRS) induced by agents such as trauma and ischemia causing extensive tissue injury to establish appropriate treatments in critically ill patients. Osteopontin acts as an extracellular matrix component or soluble cytokine in inflamed tissues. Its exact role in immune response and sepsis remains to be elucidated.Objective: This study investigated the level of osteopontin in SIRS and sepsis to assess its involvement in the acute inflammatory diseases and its possible role as a marker differentiating children with SIRS from those with sepsis.Methods: Prospective, observational study at pediatric ICU at the children’s Hospital, Zagazig University, Egypt, from October 2013 to December 2014. Fortyfour patients with SIRS or sepsis and 44 healthy subjects were enrolled. All the children were subjected to detailed medical history, Clinical examination, laboratory estimation for CBC, blood cultures, serum osteopontin and IL-6 determination was performed by sandwich enzyme immunoassay technique.Results: Serum osteopontin levels were significantly higher in patients than in controls and in sepsis than in SIRS, and decreased during the resolution of both the disorders. A receiver operating characteristic curve identified that osteopontin level of 1040 ng/ml has discriminative power between SIRS and sepsis patients with 82.6% sensitivity and 70.4% specificity, area under curve was 0.833. Osteopontin levels directly correlated interleukin-6 levels and clinical severity scores.Conclusion: Osteopontin is strongly up-regulated during SIRS and sepsis and correlate with IL6 and clinical severity scores.Keywords: Sepsis, Inflammation, Osteopontin, IL-6, Cytokine

CD44 Receptor Mediates Urate Crystal Phagocytosis by Macrophages and Regulates Inflammation in A Murine Peritoneal Model of Acute Gout

Gout is a chronic arthritis caused by the deposition of poorly soluble monosodium urate monohydrate (MSU) crystals in peripheral joints. Resident macrophages initiate inflammation in response to MSU mediated by NF-κB nuclear translocation and NLRP3 inflammasome activation. We investigated the role of CD44, a transmembrane receptor, in mediating MSU phagocytosis by macrophages. We used an antibody that sheds the extracellular domain (ECD) of CD44 to study the role of the receptor and its associated protein phosphatase 2A (PP2A) in macrophage activation. We also studied the significance of CD44 in mediating MSU inflammation in-vivo. Cd44−/− BMDMs showed reduced MSU phagocytosis, LDH release, IL-1β expression and production compared to Cd44+/+ BMDMs. Elevated CD44 staining was detected intracellularly and CD44 colocalized with α-tubulin as a result of MSU exposure and ECD-shedding reduced MSU phagocytosis in murine and human macrophages. Anti-CD44 antibody treatment reduced NF-κB p65 subunit nuclear levels, IL-1β expression, pro-IL-1β and IL-8 production in MSU stimulated THP-1 macrophages (p \u3c 0.01). The effect of the antibody was mediated by an enhancement in PP2A activity. CD44 ECD-shedding reduced the conversion of procaspase-1 to active caspase-1, caspase-1 activity and resultant generation of mature IL-1β in macrophages. Neutrophil and monocyte influx and upregulated production of IL-1β was evident in wildtype mice. MSU failed to trigger neutrophil and monocyte recruitment in Cd44−/− mice and lower IL-1β levels were detected in peritoneal lavages from Cd44−/− mice (p \u3c 0.01). Anti-CD44 antibody treatment reduced neutrophil and monocyte recruitment and resulted in reduced lavage IL-1β levels in the same model. CD44 plays a biologically significant role in mediating phagocytosis of MSU and downstream inflammation and is a novel target in gout treatment

Fingolimod Phosphate (FTY720-P) Activates Protein Phosphatase 2A in Human Monocytes and Inhibits Monosodium Urate Crystal-Induced Interleukin-1 β Production

Gout is a chronic inflammatory arthritis caused by monosodium urate monohydrate (MSU) crystal deposits in joints of lower limbs. Phagocytic uptake of MSU crystals by joint-resident macrophages and recruited circulating monocytes results in IL-1β expression and production. Current acute gout treatments have serious toxicities and suffer suboptimal clinical outcomes. Protein phosphatase 2A (PP2A) plays an important role in regulating signaling pathways relevant to inflammation. We hypothesized that innate immune danger signals, e.g., lipopolysaccharide (LPS) and soluble uric acid (sUA), prime human monocytes toward MSU crystal phagocytosis and that increased IL-1β production mediated by a reduction in PP2A activity and restoring PP2A activity exerts an anti-inflammatory effect in this setting. Priming monocytes with LPS + sUA increased cytosolic pro-IL-1β and mature IL-1β and enhanced MSU crystal phagocytosis and its downstream IL-1β expression (P \u3c 0.001). A combination of LPS + sUA priming and MSU crystals reduced PP2A activity in monocytes by 60% (P = 0.013). PP2A catalytic subunit gene knockdown reduced PP2A activity and exacerbated MSU crystal–induced IL-1β expression and secretion (P \u3c 0.0001). Fingolimod (FTY720) and its active metabolite, fingolimod phosphate (FTY720-P), were evaluated for their ability to activate PP2A in human monocytes over 24 hours. FTY720 and FTY720-P activated PP2A to a similar extent, and maximal enzyme activity occurred at 24 hours for FTY720 and at 6 hours for FTY720-P. FTY720-P (2.5 μM) reduced pro-IL-1β production and IL-1β secretion in primed and MSU crystal–stimulated monocytes (P \u3c 0.0001) without changing the magnitude of crystal phagocytosis. We conclude that PP2A is a promising new target in acute gout

Activation of Adenylyl Cyclase Reduces TGF-b Profibrotic Response in Osteoarthritic Fibroblast-like Synoviocytes

Purpose: The hallmarks of osteoarthritis (OA) include cartilage degeneration, bone remodeling and synovial fibrosis. Synovial fibrosis is characterized by excessive extracellular matrix (ECM) accumulation due to an imbalance in ECM production, in particular collagen, and its turnover. Transforming growth factor beta (TGF-β) and its associated signaling pathway mediated by ALK5, plays an important role in synovial fibrosis and blocking TGF-β’s effect prevents synovial fibrosis. Increasing intracellular cyclic AMP (cAMP) produces an antifibrotic effect in fibroblasts of multiple origins. Forskolin (FsK) is a naturally occurring diterpene in the roots of the Indian Coleus plant that activates adenylyl cyclase resulting in an elevation in intracellular cAMP levels. We hypothesized that FsK treatment results in an anti-fibrotic effect in TGF-β stimulated fibroblast-like synoviocytes (FLS) from patients with advanced OA. Methods: OA FLS (Cell Applications, USA) were harvested from patients undergoing total knee replacement. Cells were used between the 3rd and 6th passages for all experiments. OA FLS (300,000 cells per well) were treated with TGF-β (1ng/ml; R&D Systems) in the absence or presence of FsK (10μM; Sigma Aldrich) or SB431542, an ALK5 inhibitor (1μM, Sigma Aldrich) for 24 hours followed by RNA extraction using Trizol reagent and RNA concentrations were determined using a NanoDrop ND-2000 spectrophotometer. cDNA was synthesized using iScript Reverse Transcription Supermix for RT-qPCR (Bio-Rad, USA). Quantitative PCR (qPCR) was performed using TaqMan Fast Advanced Master Mix (Lifetechnologies, USA). The cycle threshold (Ct) value of genes of interest were normalized to the Ct value of GAPDH in the same sample, and the relative expression was calculated using the 2−ΔΔCt method. Genes of interest included collagens type 1 (COL1A1) and 3 (COL3A1), α2 smooth muscle actin (ACTA2), proteoglycan-4 (PRG4), matrix metalloproteinases 3, 9 and 13 (MMP3, MMP9 and MMP13), tissue inhibitor of metalloproteinase-1 (TIMP1) and aggrecanase-1 (ADAMTS4). Multiple group comparisons were performed by ANOVA or ANOVA on the ranks followed by pairwise group comparisons using Tukey\u27s test. Data is presented as the average ± S.D. of 3–6 independent experiments. Results:FsK treatment significantly reduced TGF-β induced expression of collagen type I (fig. 1A; p Conclusions: Using a model of TGF-β stimulated OA synovial fibroblasts, FsK treatment resulted in a reduction in the expression of collagen type I, a major component of fibrosis and α2 smooth muscle actin, a marker of fibroblast differentiation to myofibroblasts. To this end, FsK\u27s effect was comparable to the inhibition of intracellular TGF-β signaling. PRG4 regulates synovial proliferation and inflammation and FsK treatment enhanced PRG4 expression by OA fibroblasts. FsK reduced expression of matrix degrading enzymes, especially MMP3 and MMP9 involved in synovial proliferation, and MMP13 and ADAMTS4, involved in cartilage degradation. Increasing intracellular levels of cAMP in synovial fibroblasts may result in antifibrotic and chondroprotective effects in the joint

Design and Biological Evaluation of Colchicine-CD44-Targeted Peptide Conjugate in an In Vitro Model of Crystal Induced Inflammation

Gout is an inflammatory arthritis due to the joint deposition of monosodium urate (MSU) crystals. Phagocytosis of MSU crystals by tissue macrophages results in the generation of reactive oxygen species (ROS) and production of inflammatory cytokines and chemokines. Colchicine use in gout is limited by severe toxicity. CD44 is a transmembrane glycoprotein that is highly expressed in tissue macrophages and may be involved in gout pathogenesis. The P6 peptide is a 20-amino acid residue peptide that binds to CD44. We hypothesized that the conjugation of colchicine to the P6 peptide would reduce its off-target cytotoxicity while preserving its anti-inflammatory effect. A modified version of P6 peptide and colchicine-P6 peptide conjugate were synthesized using Fmoc/tBu solid-phase and solution-phase chemistry, respectively. A glutaryl amide was used as a linker. The P6 peptide was evaluated for its binding to CD44, association, and internalization by macrophages. Cytotoxic effects of P6 peptide, colchicine, and colchicine-P6 peptide on macrophages were compared and the inhibition of ROS generation and interleukin-8 (IL-8) secretion in MSU-stimulated macrophages treated with P6 peptide, colchicine, or colchicine-P6 peptide was studied. We confirmed that the P6 peptide binds to CD44 and its association and internalization by macrophages were CD44-dependent. Colchicine (1, 10, and 25 µM) demonstrated a significant cytotoxic effect on macrophages while the P6 peptide and colchicine-P6 peptide conjugate (1, 10 and 25 µM) did not alter the viability of the macrophages. The P6 peptide (10 and 25 µM) reduced ROS generation and IL-8 secretion mediated by a reduction in MSU phagocytosis by macrophages. The colchicine-P6 peptide significantly reduced ROS generation and IL-8 secretion compared to the P6 peptide alone at 1 and 10 µM concentrations. Conjugation of colchicine to the P6 peptide reduced the cytotoxic effect of colchicine while preserving its anti-inflammatory activity