Backpropagating through Structured Argmax using a SPIGOT

We introduce the structured projection of intermediate gradients optimization technique (SPIGOT), a new method for backpropagating through neural networks that include hard-decision structured predictions (e.g., parsing) in intermediate layers. SPIGOT requires no marginal inference, unlike structured attention networks (Kim et al., 2017) and some reinforcement learning-inspired solutions (Yogatama et al., 2017). Like so-called straight-through estimators (Hinton, 2012), SPIGOT defines gradient-like quantities associated with intermediate nondifferentiable operations, allowing backpropagation before and after them; SPIGOT's proxy aims to ensure that, after a parameter update, the intermediate structure will remain well-formed. We experiment on two structured NLP pipelines: syntactic-then-semantic dependency parsing, and semantic parsing followed by sentiment classification. We show that training with SPIGOT leads to a larger improvement on the downstream task than a modularly-trained pipeline, the straight-through estimator, and structured attention, reaching a new state of the art on semantic dependency parsing.Comment: ACL 201

DiFX2: A more flexible, efficient, robust and powerful software correlator

Software correlation, where a correlation algorithm written in a high-level language such as C++ is run on commodity computer hardware, has become increasingly attractive for small to medium sized and/or bandwidth constrained radio interferometers. In particular, many long baseline arrays (which typically have fewer than 20 elements and are restricted in observing bandwidth by costly recording hardware and media) have utilized software correlators for rapid, cost-effective correlator upgrades to allow compatibility with new, wider bandwidth recording systems and improve correlator flexibility. The DiFX correlator, made publicly available in 2007, has been a popular choice in such upgrades and is now used for production correlation by a number of observatories and research groups worldwide. Here we describe the evolution in the capabilities of the DiFX correlator over the past three years, including a number of new capabilities, substantial performance improvements, and a large amount of supporting infrastructure to ease use of the code. New capabilities include the ability to correlate a large number of phase centers in a single correlation pass, the extraction of phase calibration tones, correlation of disparate but overlapping sub-bands, the production of rapidly sampled filterbank and kurtosis data at minimal cost, and many more. The latest version of the code is at least 15% faster than the original, and in certain situations many times this value. Finally, we also present detailed test results validating the correctness of the new code.Comment: 28 pages, 9 figures, accepted for publication in PAS

Zuverlässige numerische Berechnungen mit dem Spigot-Ansatz

Der Spigot-Ansatz ist eine elegante Vorgehensweise, spezielle numerische Werte zuverlässig, effizient und mit beliebiger Genauigkeit zu berechnen. Die Stärke des Ansatzes ist seine Effizienz, seine totale Korrektheit und seine mathematisch exakt begründete Sicherstellung einer gewünschten absoluten Genauigkeit. Seine Schwäche ist möglicherweise die eingeschränkte Anwendbarkeit. Es gibt in der Literatur Spigot-Berechnung für e und pi. Wurzelberechnung und Logarithmenberechnung gehören zu den Hauptergebnissen der Dissertation. In Kombination mit den Methoden der Reihentransformation von Zeilberger und Wilf bzw. von Gosper ist der Einsatz zur Berechnung von hypergeometrischen Reihen sehr Erfolg versprechend. Eine interessante offene Frage ist die Berechnung der Feigenbaumkonstanten mit dem Ansatz. 'Spigot' bedeutet 'sukzessive Extraktion von Wertanteilen': die Wertanteile werden extrahiert, als ob sie durch einen Hahn (englisch: spigot) gepumpt werden. Es ist dabei besonders interessant, dass in bestimmten Fällen ein Wert-Anteil mit einer Ziffer der Kodierung des Ergebnisses versehen werden kann. Der Spigot-Ansatz steht damit im Gegensatz zu den konventionellen Iterationsverfahren: in einem Schritt des Spigot-Ansatzes wird jeweils ein Wert-Anteil 'extrahiert' und das gesamte Ergebnis ist die Summe der Wert-Anteile; während ein Schritt in einem Iterationsverfahren die Berechnung eines besseren gesamten Ergebnisses aus dem des vorigen Schritt beinhaltet. Das Grundschema der Berechnung mit dem Spigot-Ansatz sieht folgendermaßen aus: zuerst wird für den zu berechnenden numerischen Wert eine gut konvergierende Reihe mit rationalen Gliedern durch symbolisch-algebraische Methoden hergeleitet; dann wird für eine gewünschte Genauigkeit eine Teilsumme ausgewählt; anschließend werden aus der Teilsumme Wert-Anteile iterativ extrahiert. Die Extraktion von Wert-Anteilen aus der Teilsumme geschieht mit dem Spigot-Algorithmus, der auf Sale zurück geht, nur Integer-Arithmetik benötigt und sich als eine verallgemeinerte Form der Basis-Konvertierung dadurch auffassen lässt, dass die Teilsumme als die Kodierung des Wertes in einer inhomogenen Basis interpretiert wird. Die Spigot-Idee findet auch in der Überführung einer Reihe in eine besser konvergierende Reihe auf der Art und Weise Anwendung, dass Wert-Anteile aus der Reihe extrahiert werden, um die originale Reihe werttreu zur Reihe der Wert-Anteile zu transformieren. Dies geschieht mit den Methoden der Reihentransformation von Gosper bzw. Wilf. Die Dissertation umfasst im Wesentlichen die Formalisierung des Spigot-Algorithmus und der Gosperschen Reihentransformation, eine systematische Darstellung der Ansätze, Methoden und Techniken der Reihenentwiclung und Reihentransformation (die Herleitung von Reihen mit Hilfe charakteristischer Funktionalgleichungen; Methoden der Reihentransformation von Euler, Kummer, Markoff, Gosper, Zeilberger und Wilf) sowie die Methoden zur Berechnung von Wurzeln und Logarithmen mit dem Spigot-Ansatz. Es ist interessant zu sehen, wie sich die Grundideen des Spigot-Algorithmus, der Wurzelberechnung und der Logarithmenberechnung jeweils im Wesentlichen durch eine Gleichung ausdrücken lassen. Es ist auch interessant zu sehen, wie sich verschiedene Methoden der Reihentransformation auf einige einfache Grundideen zurückführen lassen. Beispiele für den Beweis von totalen Korrektheit (bei iterativer Berechnung von Wurzeln) könnte auch von starkem Interesse sein. Um die Zuverlässigkeit anderer Methoden zur Berechnung von natürlichen Logarithmen zu überprüfen, scheint der Vergleich der Ergebnisse mit den des Spigot-Ansatzes die beste Methode zu sein. Anders als bei Wurzelberechnung kann hierbei zur Überprüfung die inverse Berechnung nicht angewandt werden.spigot, total correctness, acceleration of series, computation of roots, computation of logarithms Reliable numerical computations with spigot approach Spigot approach is an elegant way to compute special numerical values reliably, efficiently and with arbitrary accuracy. The advantage of this way are its efficiency and its total correctness including the bounding of the absolute error. The disadvantage is perhaps its restricted applicableness. There are spigot computation for e an pi. The computation of roots and logarithms belongs to the main results of this thesis. In combination with the methods for acceleration of series by Gosper as well as by Zeilberger and Wilf is the use for numerical summation of hypergeometric series very promising. An interesting open question is the computation of the Feigenbaum constant by this way. ‘Spigot’ means ‘successive extraction of portions of value’: the portions of value are ‘extracted’ as if they were pumped through a spigot. It is very interesting in certain case, where these portions can be interpreted as the digits of the result. With respect to that the spigot approach is the opposition to the iterative approach, where in each step the new better result is computed from the result of the previous step. The schema of spigot approach is characterised as follows: first a series for the value to be computed is derived, then a partial sum of the series is chosen with respect to an desired accuracy, afterwards the portions of value are extracted from the sum. The extraction of potions of value is carried by the spigot algorithm which is due to Sale an requires only integer arithmetic. The spigot algorithm can be understood as a generalisation of radix-conversion if the sum is interpreted as an encoding of the result in a mixed-radix (inhomogeneous) system. The spigot idea is also applied in transferring a series into a better convergent series: portions of value are extracted successively from the original series in order to form the series of extracted potions which should have the same value as the original series. This transfer is carried with the methods for acceleration of series by Gosper and Wilf. The thesis incorporates essentially the formalisation of the spigot algorithm and the method of Gosper for acceleration of series, a systematisation of methods and techniques for derivation and acceleration of series (derivation of series for functions by using their characteristic functional equations; methods for acceleration of series by Euler, Kummer, Markov, Gosper Zeilberger and Wilf) as well as the methods for computation of roots and logarithms by spigot approach. It is interesting to see how to express the basic ideas for spigot algorithm, computation of roots and computation of logarithm respectively in some equations. It is also interesting to see how to build various methods for acceleration of series from some simple basic ideas. Examples for proof of total correctness (for iterative computation of roots) can be of value to read. Comparing with the results produced by spigot approach is possibly the best way for verifying the reliability of other methods for computation of natural logarithms, because (as opposed to root computing) the verification by inverse computation is inapplicable

Tolerance analysis and synthesis of assemblies subject to loading with process integration and design optimization tools

Manufacturing variation results in uncertainty in the functionality and performance of mechanical assemblies. Management of this uncertainty is of paramount importance for manufacturing efficiency. Methods focused on the management of uncertainty and variation in the design of mechanical assemblies, such as tolerance analysis and synthesis, have been subject to extensive research and development to date. However, due to the challenges involved, limitations in the capability of these methods remain. These limitations are associated with the following problems: The identification of Key Product Characteristics (KPCs) in mechanical assemblies (which are required for measuring functional performance) without imposing significant modelling demands.  Accommodation of the high computational cost of traditional statistical tolerance analysis in early design where analysis budgets are limited. Efficient identification of feasible regions and optimum performance within the large design spaces associated with early design stages.  The ability to comprehensively accommodate tolerance analysis problems in which assembly functionality is dependent on the effects of loading (such as compliance or multi‐body dynamics). Current Computer Aided Tolerancing (CAT) is limited by: the ability to accommodate only specific loading effects; reliance on custom simulation codes with limited practical implementation in accessible software tools; and, the need for additional expertise in formulating specific assembly tolerance models and interpreting results. Accommodation of the often impractically high computational cost of tolerance synthesis involving demanding assembly models (particularly assemblies under loading). The high computational cost is associated with traditional statistical tolerancing Uncertainty Quantification (UQ) methods reliant on low‐efficiency Monte Carlo (MC) sampling. This research is focused on addressing these limitations, by developing novel methods for enhancing the engineering design of mechanical assemblies involving uncertainty or variation in design parameters. This is achieved by utilising the emerging design analysis and refinement capabilities of Process Integration and Design Optimization (PIDO) tools. ii The main contributions of this research are in three main themes:  Design analysis and refinement accommodating uncertainty in early design;  Tolerancing of assemblies subject to loading; and, efficient Uncertainty Quantification (UQ) in tolerance analysis and synthesis. The research outcomes present a number of contributions within each research theme, as outlined below. Design analysis and refinement accommodating uncertainty in early design: A PIDO tool based visualization method to aid designers in identifying assembly KPCs in early design stages. The developed method integrates CAD software functionally with the process integration, UQ, data logging and statistical analysis capabilities of PIDO tools, to simulate manufacturing variation in an assembly and visualise assembly clearances, contacts or interferences. The visualization capability subsequently assists the designer in specifying critical assembly dimensions as KPCs.  Computationally efficient method for manufacturing sensitivity analysis of assemblies with linear‐compliant elements. Reduction in computational cost are achieved by utilising linear‐compliant assembly stiffness measures, reuse of CAD models created in early design stages, and PIDO tool based tolerance analysis. The associated increase in computational efficiency, allows an estimate of sensitivity to manufacturing variation to be made earlier in the design process with low effort.  Refinement of concept design embodiments through PIDO based DOE analysis and optimization. PIDO tools are utilised to allow CAE tool integration, and efficient reuse of models created in early design stages, to rapidly identify feasible and optimal regions in the design space. A case study focused on the conceptual design of automotive seat kinematics is presented, in which an optimal design is identified and subsequently selected for commercialisation in the Tesla Motors Model S full‐sized electric sedan. These contributions can be directly applied to improve the design of mechanical assemblies involving uncertainty or variation in design parameters in the early stages of design. The use of native CAD/E models developed as part of an established design modelling procedure imposes low additional modelling effort. Tolerancing of assemblies subject to loading:  A novel tolerance analysis platform is developed which integrates CAD/E and statistical analysis tools using PIDO tool capabilities to facilitate tolerance analysis of assemblies subject to loading. The proposed platform extends the capabilities of traditional CAT tools and methods by enabling tolerance analysis of assemblies which are dependent on iii the effects of loads. The ability to accommodate the effects of loading in tolerance analysis allows for an increased level of capability in estimating the effects of variation on functionality.  The interdisciplinary integration capabilities of the PIDO based platform allow for CAD/E models created as part of the standard design process to be used for tolerance analysis. The need for additional modelling tools and expertise is subsequently reduced.  Application of the developed platform resulted in effective solutions to practical, industry based tolerance analysis problems, including: an automotive actuator mechanism assembly consisting of rigid and compliant components subject to external forces; and a rotary switch and spring loaded radial detent assembly in which functionality is defined by external forces and internal multi‐body dynamics. In both case studies the tolerance analysis platform was applied to specify nominal dimensions and required tolerances to achieve the desired assembly yield. The computational platform offers an accessible tolerance analysis approach for accommodating assemblies subject to loading with low implementation demands. Efficient Uncertainty Quantification (UQ) in tolerance analysis and synthesis:  A novel approach is developed for addressing the high computational cost of Monte Carlo (MC) sampling in statistical tolerance analysis and synthesis, with Polynomial Chaos Expansion (PCE) uncertainty quantification. Compared to MC sampling, PCE offers significantly higher efficiency. The feasibility of PCE based UQ in tolerance synthesis is established through: theoretical analysis of the PCE method identifying working principles, implementation requirements, advantages and limitations; identification of a preferred method for determining PCE expansion coefficients in tolerance analysis; and, formulation of an approach for the validation of PCE statistical moment estimates.  PCE based UQ is subsequently implemented in a PIDO based tolerance synthesis platform for assemblies subject to loading. The resultant PIDO based tolerance synthesis platform integrates: highly efficient sparse grid based PCE UQ, parametric CAD/E models accommodating the effects of loading, cost‐tolerance modelling, yield quantification with Process Capability Indices (PCI), optimization of tolerance cost and yield with multiobjective Genetic Algorithm (GA).  To demonstrate the capabilities of the developed platform, two industry based case studies are used for validation, including: an automotive seat rail assembly consisting of compliant components subject to loading; and an automotive switch in assembly in which functionality is defined by external forces and multi‐body dynamics. In both case studies optimal tolerances were identified which satisfied desired yield and tolerance cost objectives. The addition of PCE to the tolerance synthesis platform resulted in large computational cost reductions without compromising accuracy compared to traditional MC methods. With traditional MC sampling UQ the required computational expense is impractically high. The resulting tolerance synthesis platform can be applied to tolerance analysis and synthesis with significantly reduced computation time while maintaining accurac