Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations

Using weight decay to penalize the L2 norms of weights in neural networks has been a standard training practice to regularize the complexity of networks. In this paper, we show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with positively homogeneous activation functions, such as linear, ReLU and max-pooling functions. As a result of homogeneity, functions specified by the networks are invariant to the shifting of weight scales between layers. The ineffective regularizers are sensitive to such shifting and thus poorly regularize the model capacity, leading to overfitting. To address this shortcoming, we propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network. The derived regularizer is an upper bound for the input gradient of the network so minimizing the improved regularizer also benefits the adversarial robustness. Residual connections are also considered and we show that our regularizer also forms an upper bound to input gradients of such a residual network. We demonstrate the efficacy of our proposed regularizer on various datasets and neural network architectures at improving generalization and adversarial robustness.Comment: 14 pages, 5 figure

Bayesian Nested Neural Networks for Uncertainty Calibration and Adaptive Compression

Nested networks or slimmable networks are neural networks whose architectures can be adjusted instantly during testing time, e.g., based on computational constraints. Recent studies have focused on a "nested dropout" layer, which is able to order the nodes of a layer by importance during training, thus generating a nested set of sub-networks that are optimal for different configurations of resources. However, the dropout rate is fixed as a hyper-parameter over different layers during the whole training process. Therefore, when nodes are removed, the performance decays in a human-specified trajectory rather than in a trajectory learned from data. Another drawback is the generated sub-networks are deterministic networks without well-calibrated uncertainty. To address these two problems, we develop a Bayesian approach to nested neural networks. We propose a variational ordering unit that draws samples for nested dropout at a low cost, from a proposed Downhill distribution, which provides useful gradients to the parameters of nested dropout. Based on this approach, we design a Bayesian nested neural network that learns the order knowledge of the node distributions. In experiments, we show that the proposed approach outperforms the nested network in terms of accuracy, calibration, and out-of-domain detection in classification tasks. It also outperforms the related approach on uncertainty-critical tasks in computer vision.Comment: 16 pages, 10 figure

An inverted J-shaped association of serum uric acid with muscle strength among Japanese adult men: a cross-sectional study

BACKGROUND: Uric acid (UA) may protect muscle function from oxidative damage due to reactive oxygen species through its powerful antioxidant capacity. However, several studies have demonstrated that hyperuricemia is closely related to systemic inflammation and has oxidant properties effects, both of which may increase the risk of muscle strength loss. The purpose of this study was to examine the association of serum UA concentration with grip strength and leg extension power in adult men. METHODS: This study is a cross-sectional survey in which 630 Japanese male employees aged 30 years and older participated. Five hundred and eighty-six subjects participated in the measurement of grip strength, and 355 subjects participated in the measurement of leg extension power. Blood samples were obtained for serum UA analysis. RESULTS: After adjustment for potential confounders, grip strength differed significantly between participants with and those without hyperuricemia (geometric mean and 95% confidence interval [CI]: 40.3 [39.2–41.3] kg vs. 41.9 [41.3–42.5] kg; P = 0.01). In addition, serum UA levels (quartiles) showed an inverted J-shaped curve with grip strength (mean and 95% CI: Q1, 41.6 [40.6–42.6] kg; Q2, 42.2 [41.2–43.2] kg; Q3, 41.8 [40.8–42.8] kg; Q4, 40.4 [39.3–41.4] kg; P for quadratic trend = 0.05). The results in the leg extension power group were similar to those observed in the grip strength group. CONCLUSION: This population-based cross-sectional study shows for the first time that hyperuricemia is associated with poor muscle strength. Moreover, the results indicate an inverted J-shaped association between serum UA quartiles and muscle strength

Changes of predominant species/biovars and sequence types of Brucellaisolates, Inner Mongolia, China

BACKGROUND: Human brucellosis incidence in China was divided into 3 stages, high incidence (1950-1960s), decline (1970-1980s) and re-emergence (1990-2000s). Human brucellosis has been reported in all the 32 provinces, of which Inner Mongolia has the highest prevalence, accounting for over 40% of the cases in China. To investigate the etiology alteration of human brucellosis in Inner Mongolia, the species, biovars and genotypes of 60 Brucella isolates from this province were analyzed. METHODS: Species and biovars of the Brucella strains isolated from outbreaks were determined based on classical identification procedures. Strains were genotyped by multi locus sequence typing (MLST). Sequences of 9 housekeeping genes were obtained and sequence types were defined. The distribution of species, biovars and sequence types (STs) among the three incidence stages were analyzed and compared. RESULTS: The three stages of high incidence, decline and re-emergence were predominated by B. melitensis biovar 2 and 3, B. abortus biovar 3, and B. melitensis biovar 1, respectively, implying changes in the predominant biovars. Genotyping by MLST revealed a total of 14 STs. Nine STs (from ST28 to ST36), accounting for 64.3% of all the STs, were newly defined and different from those observed in other countries. Different STs were distributed among the three stages. ST8 was the most common ST in 1950-1960s and 1990-2000s, while ST2 was the most common in 1970-1980s. CONCLUSIONS: The prevalence of biovars and sequence types of Brucella strains from Inner Mongolia has changed over time in the three stages. Compared with those from other countries, new sequence types of Brucella strains exist in China

Euryale Ferox Seed-inspired Super-lubricated Nanoparticles for Treatment of Osteoarthritis

Osteoarthritis has been regarded as a typical lubrication deficiency related joint disease, which is characterized by the breakdown of articular cartilage at the joint surface and the inflammation of the joint capsule. Here, inspired by the structure of the fresh euryale ferox seed that possesses a slippery aril and a hard coat containing starchy kernel, a novel superlubricated nanoparticle, namely poly (3‐sulfopropyl methacrylate potassium salt)‐grafted mesoporous silica nanoparticles (MSNs‐NH2@PSPMK), is biomimicked and synthesized via a one‐step photopolymerization method. The nanoparticles are endowed with enhanced lubrication by the grafted PSPMK polyelectrolyte polymer due to the formation of tenacious hydration layers surrounding the negative charges, and simultaneously are featured with effective drug loading and release behavior as a result of the sufficient mesoporous channels in the MSNs. When encapsulated with an anti‐inflammatory drug diclofenac sodium (DS), the lubrication capability of the superlubricated nanoparticles is improved, while the drug release rate is sustained by increasing the thickness of PSPMK layer, which is simply achieved via adjustment of the precursor monomer concentration in the photopolymerization process. Additionally, the in vitro and in vivo experimental results show that the DS‐loaded MSNs‐NH2@PSPMK nanoparticles effectively protect the chondrocytes from degeneration, and thus, inhibit the development of osteoarthritis.Peer reviewe

Droplet spatial distribution of oil-based emulsion spray

IntroductionOil-based emulsion solution is a common pesticide formulation in agricultural spraying, and its spray characteristics are different from that of water spraying. The well understanding of its spray characteristics is the theoretical basis to improve the pesticide spraying technology. The objective of the present study is to deepen the understanding of the spray characteristics of oil-based emulsion.MethodIn this paper, the spatial distribution characteristics of spray droplets of oil-based emulsion were captured visually using the high-speed photomicrography. On the basis of image processing method, the droplet size and distribution density of spray droplets at different spatial locations were analyzed quantitatively. The effects of nozzle configuration and emulsion concentration on spray structures and droplet spatial distribution were discussed.ResultsOil-based emulsion produced a special perforation atomization mechanism compared with water spray, which led to the increase of spray droplet size and distribution density. Nozzle configuration had a significant effect on oil-based emulsion spray, with the nozzle changed from ST110-01 to ST110-03 and ST110-05; the sheet lengths increased to 18 and 28 mm, respectively, whereas the volumetric median diameters increased to 51.19% and 76.00%, respectively. With emulsion concentration increased from 0.02% to 0.1% and 0.5%, the volumetric median diameters increased to 5.17% and 14.56%, respectively.DiscussionThe spray droplet size of oil-based emulsion spray can be scaled by the equivalent diameter of discharge orifice of nozzles. The products of volumetric median diameters and corresponding surface tensions were nearly constant for the oil-based emulsion spray of different emulsion concentrations. It is expected that this research could provide theoretical support for improving the spraying technology of oil-based emulsion and increasing the utilization of pesticide

Correlating Chemical Reaction and Mass Transport in Hydrogen-based Direct Reduction of Iron Oxide

Steelmaking contributes 8% to the total CO2 emissions globally, primarily due to coal-based iron ore reduction. Clean hydrogen-based ironmaking has variable performance because the dominant gas-solid reduction mechanism is set by the defects and pores inside the mm-nm sized oxide particles that change significantly as the reaction progresses. While these governing dynamics are essential to establish continuous flow of iron and its ores through reactors, the direct link between agglomeration and chemistry is still contested due to missing measurements. In this work, we directly measure the connection between chemistry and agglomeration in the smallest iron oxides relevant to magnetite ores. Using synthesized spherical 10-nm magnetite particles reacting in H2, we resolve the formation and consumption of w\"ustite (FeO) - the step most commonly attributed to agglomeration. Using X-ray scattering and microscopy, we resolve crystallographic anisotropy in the rate of the initial reaction, which becomes isotropic as the material sinters. Complementing with imaging, we demonstrate how the particles self-assemble, subsequently react and sinter into ~100x oblong grains. Our insights into how morphologically uniform iron oxide particles react and agglomerate H2 reduction enable future size-dependent models to effectively describe the multiscale iron ore reduction

Skin advanced glycation end product accumulation and muscle strength among adult men

Aging is associated with decreased skeletal muscle function. Increased levels of advanced glycation end products (AGEs) in skeletal muscle tissue are observed with advancing age and in diabetes. Although serum AGE level is negatively associated with grip strength in elderly people, it is unknown whether this association is present in adult males. To determine the relationship between AGE accumulation in tissue and muscle strength and power among Japanese adult men. Skin autofluorescence (AF) (a noninvasive method for measuring tissue AGEs), grip strength (n = 232), and leg extension power (n = 138) were measured in Japanese adult men [median (interquartile range) age, 46.0 (37.0, 56.0) years]. After adjustment for potential confounders, the adjusted means [95% confidence interval (CI)] for grip strength across the tertiles of skin AF were 44.5 (43.2, 45.9) kg for the lowest tertile, 42.0 (40.6, 43.3) kg for the middle tertile, and 41.7 (40.3, 43.1) kg for the highest tertile (P for trend < 0.01). Moreover, the adjusted geometric means (95% CI) of leg extension power across the tertiles of skin AF were 17.8 (16.6, 19.1) W/kg for the lowest tertile, 17.5 (16.4, 18.7) W/kg for the middle tertile, and 16.0 (14.9, 17.1) W/kg for the highest tertile (P for trend = 0.04). Among Japanese adult men, participants with higher skin AF had lower muscle strength and power, indicating a relationship between AGE accumulation and muscle strength and power. A long-term prospective study is required to clarify the causality