How degenerate is the parametrization of neural networks with the ReLU activation function?

Neural network training is usually accomplished by solving a non-convex optimization problem using stochastic gradient descent. Although one optimizes over the networks parameters, the main loss function generally only depends on the realization of the neural network, i.e. the function it computes. Studying the optimization problem over the space of realizations opens up new ways to understand neural network training. In particular, usual loss functions like mean squared error and categorical cross entropy are convex on spaces of neural network realizations, which themselves are non-convex. Approximation capabilities of neural networks can be used to deal with the latter non-convexity, which allows us to establish that for sufficiently large networks local minima of a regularized optimization problem on the realization space are almost optimal. Note, however, that each realization has many different, possibly degenerate, parametrizations. In particular, a local minimum in the parametrization space needs not correspond to a local minimum in the realization space. To establish such a connection, inverse stability of the realization map is required, meaning that proximity of realizations must imply proximity of corresponding parametrizations. We present pathologies which prevent inverse stability in general, and, for shallow networks, proceed to establish a restricted space of parametrizations on which we have inverse stability w.r.t. to a Sobolev norm. Furthermore, we show that by optimizing over such restricted sets, it is still possible to learn any function which can be learned by optimization over unrestricted sets.Comment: Accepted at NeurIPS 201

Begrenzte Heterotrophie bei Amphidinium (Dinoflagellata)

An bakterienfreien Kulturen von Amphidinium höfleri in vollsynthetischem Nährmedium konnte mit dem Coulter-Counter gezeigt werden, daß gelöste Aminosäuren in Konzentrationen von 50 mg/l eine Erhöhung des Zellvolumens der Population um 6,1% bewirken. Eine Beeinflussung der Generationszeit konnte nicht festgestellt werden. Wurden die Aminosäuren in Konzentrationen von 200 µg/l, also in Konzentrationen, wie sie im Pelagial des Meeres vorkommen, geboten, so konnte keine Förderung des Wachstums nachgewiesen werden. With axenic cultures of Amphidinium höfleri it was shown by using the Coulter-Counter, that dissolved amino-acids in concentrations of 50 mg/l will increase the cellvolume by 6.1%. There was no effect on the generation time. If the aminoacids were added in concentrations of 200 µg/l - the concentration of natural sea water - then there was no effect

A Gordian knot: Nomenclature and taxonomy of Heterocapsa triquetra (Peridiniales: Heterocapsaceae)

In the course of polyphasic taxonomic work in the dinophytes, we became aware of a fundamental misapplication of the name Glenodinium triquetrum (now represented conceptually by a species of Kryptoperidinium), when Stein assigned it to Heterocapsa. Possible solutions involve a conflict between retaining Ehrenberg’s epithet in its correct application in the interest of priority and preserving current usage of Heterocapsa. However, we do not achieve a consensus on how to disentangle this Gordian knot, underlining that this is not a regular case of taxonomic confusion. We intend to stimulate a more general discussion about best practices in such cases, balancing between the interest of nomenclatural practicability and the respectful acknowledgement of scientific work, even if it was conducted many years ago

Sexual reproduction and two different encystment strategies of Lingulodinium polyedrum (Dinophyceae) in culture

Unreported aspects in the sexual cycle of the marine dinoflagellate Lingulodinium polyedrum (Stein) Dodge were described. Our observations included the description of two types of hypnozygote formation, because culture planozygotes were observed to encyst in two different ways: an ecdysal sexual stage or a spiny resting cyst. Phosphate deficiency was the main nutritional condition required for fusing gamete pairs to form resting cysts, whereas replete conditions prevented their appearance and favored the formation of ecdysal sexual forms. Mating experiments revealed the existence of two sexual types (+/−), which were enough to explain resting cyst appearance (simple heterothallism). Morphological aspects and timing of gamete mating, fusion, and the efficiency of encystment under different external levels of nitrate and phosphate were analyzed after isolating and monitoring individual pairs of fusing gametes. The staining of sexual stages showed that nuclear fusion was completed at the same time as the cytoplasmic fusion. After 1 to 2 h, the planozygotes presented one quadrolobulated nucleus. Germination of ecdysal sexual stages occurred after <24–72 h, whereas excystment of resting cysts was dependent on the studied parental cross and took place after 2–4 months. Newly germinated cells from both types of cysts had a similar, big, U-shaped nucleus. Twenty-four to 48 h after excystment, the germlings divided by desmoschisis, a process before which enlargement of the nucleus was observed.Postprin

An Updated List of Generic Names in the Thoracosphaeraceae

Calcareous dinophytes produce exoskeletal calcified structures during their life history (a unique character among the alveolates) and are subsumed under the Thoracosphaeraceae as part of the Peridiniales. We provide a brief synopsis about the taxonomic history of the group, from the first descriptions of fossils in the 19th century through to the results of molecular phylogenetics studies undertaken during the past two decades. Delimitation and circumscription of the Thoracosphaeraceae are challenging, as they comprise both phototrophic (presumably including endosymbiotic) as well as heterotrophic (and even parasitic) dinophytes from marine and freshwater environments, respectively. However, calcareous structures are not known from all members of the Thoracosphaeraceae, and the corresponding species and groups are considered to have lost the capacity to calcify. Five years ago, a taxonomic list of 99 generic names assigned to the Thoracosphaeraceae was published, and we update this compendium with 19 additional names based on recent studies