Activations and Gradients Compression for Model-Parallel Training

Large neural networks require enormous computational clusters of machines. Model-parallel training, when the model architecture is partitioned sequentially between workers, is a popular approach for training modern models. Information compression can be applied to decrease workers communication time, as it is often a bottleneck in such systems. This work explores how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques. Moreover, we employ TopK with AQ-SGD per-batch error feedback approach. We conduct experiments on image classification and language model fine-tuning tasks. Our findings demonstrate that gradients require milder compression rates than activations. We observe that $K=10\%$ is the lowest TopK compression level, which does not harm model convergence severely. Experiments also show that models trained with TopK perform well only when compression is also applied during inference. We find that error feedback techniques do not improve model-parallel training compared to plain compression, but allow model inference without compression with almost no quality drop. Finally, when applied with the AQ-SGD approach, TopK stronger than with $K=30\%$ worsens model performance significantly.Comment: 17 pages, 6 figures, 5 table

Fluctuating Nonlinear Spring Model of Mechanical Deformation of Biological Particles

We present a new theory for modeling forced indentation spectral lineshapes of biological particles, which considers non-linear Hertzian deformation due to an indenter-particle physical contact and bending deformations of curved beams modeling the particle structure. The bending of beams beyond the critical point triggers the particle dynamic transition to the collapsed state, an extreme event leading to the catastrophic force drop as observed in the force (F)-deformation (X) spectra. The theory interprets fine features of the spectra: the slope of the FX curves and the position of force-peak signal, in terms of mechanical characteristics --- the Young's moduli for Hertzian and bending deformations E_H and E_b, and the probability distribution of the maximum strength with the strength of the strongest beam F_b^* and the beams' failure rate m. The theory is applied to successfully characterize the $FX$ curves for spherical virus particles --- CCMV, TrV, and AdV

Tubulin bond energies and microtubule biomechanics determined from nanoindentation in silico

Microtubules, the primary components of the chromosome segregation machinery, are stabilized by longitudinal and lateral non-covalent bonds between the tubulin subunits. However, the thermodynamics of these bonds and the microtubule physico-chemical properties are poorly understood. Here, we explore the biomechanics of microtubule polymers using multiscale computational modeling and nanoindentations in silico of a contiguous microtubule fragment. A close match between the simulated and experimental force-deformation spectra enabled us to correlate the microtubule biomechanics with dynamic structural transitions at the nanoscale. Our mechanical testing revealed that the compressed MT behaves as a system of rigid elements interconnected through a network of lateral and longitudinal elastic bonds. The initial regime of continuous elastic deformation of the microtubule is followed by the transition regime, during which the microtubule lattice undergoes discrete structural changes, which include first the reversible dissociation of lateral bonds followed by irreversible dissociation of the longitudinal bonds. We have determined the free energies of dissociation of the lateral (6.9+/-0.4 kcal/mol) and longitudinal (14.9+/-1.5 kcal/mol) tubulin-tubulin bonds. These values in conjunction with the large flexural rigidity of tubulin protofilaments obtained (18,000-26,000 pN*nm^2), support the idea that the disassembling microtubule is capable of generating a large mechanical force to move chromosomes during cell division. Our computational modeling offers a comprehensive quantitative platform to link molecular tubulin characteristics with the physiological behavior of microtubules. The developed in silico nanoindentation method provides a powerful tool for the exploration of biomechanical properties of other cytoskeletal and multiprotein assemblie

Protein–protein docking by fast generalized Fourier transforms on 5D rotational manifolds

International audienceEnergy evaluation using fast Fourier transforms (FFTs) enables sampling billions of putative complex structures and hence revolutionized rigid protein–protein docking. However, in current methods, efficient acceleration is achieved only in either the translational or the rotational subspace. Developing an efficient and accurate docking method that expands FFT-based sampling to five rotational coordinates is an extensively studied but still unsolved problem. The algorithm presented here retains the accuracy of earlier methods but yields at least 10-fold speedup. The improvement is due to two innovations. First, the search space is treated as the product manifold SO(3)×(SO(3)∖S1), where SO(3) is the rotation group representing the space of the rotating ligand, and (SO(3)∖S1) is the space spanned by the two Euler angles that define the orientation of the vector from the center of the fixed receptor toward the center of the ligand. This representation enables the use of efficient FFT methods developed for SO(3). Second, we select the centers of highly populated clusters of docked structures, rather than the lowest energy conformations, as predictions of the complex, and hence there is no need for very high accuracy in energy evaluation. Therefore, it is sufficient to use a limited number of spherical basis functions in the Fourier space, which increases the efficiency of sampling while retaining the accuracy of docking results. A major advantage of the method is that, in contrast to classical approaches, increasing the number of correlation function terms is computationally inexpensive, which enables using complex energy functions for scoring

Generalization Second Order Macroscopic Traffic Models via Relative Velocity of the Congestion Propagation

This paper presents a generalized second-order hydrodynamic traffic model. Its central piece is the expression for the relative velocity of the congestion (compression wave) propagation. We show that the well-known second-order models of Payne–Whitham, Aw–Rascal and Zhang are all special cases of the featured generalized model, and their properties are fully defined by how the relative velocity of the congestion is expressed. The proposed model is verified with traffic data from a segment of the Interstate 580 freeway in California, USA, collected by the California DOT’s Performance Measurement System (PeMS)

Evaluation of public transport fare policy using smartcard data: Travel patterns change and distributional effects in Stockholm County

This study investigated the potential of ex-post fare policy evaluation utilising smartcard data. It looked into how to identify and measure the actual outcomes of a policy, define the degree of its success, and even find some unexpected side-effects of high importance. The study is based on a case of the public transport fare policy introduced by the regional administration of Stockholm County in January, 2017. The policy focused on changing the fare structure basis, in particular switching from a zonal to a flat-fare scheme. The goal of the study is to determine how the policy affected the traveling behavior of different public transport user groups and what equity implications it brought. Altogether, these insights could be transformed into lessons for designing a more advanced evaluation approach and future policy-making.FairAccess. SLL-KTH research projec

An Evolutionary View on Equilibrium Models of Transport Flows

In this short paper, we describe natural logit population games dynamics that explain equilibrium models of origin-destination matrix estimation and (stochastic) traffic assignment models (Beckmann, Nesterov–de Palma). Composition of the proposed dynamics allows to explain two-stages traffic assignment models

How fair is the fare? Estimating travel patterns and the impacts of fare schemes for different user groups in Stockholm based on smartcard data : Final report for Trafik och Region 2018 SLL-KTH research project

There is a rapid increase in the deployment, acquisition and analysis of automated fare collection (AFC) systems, enabling a profound change in the ability to analyze high-volume data that relate to observed passenger travel behavior and recurrent patterns. The analysis of such passively collected data offers direct access to a continuous flow of observed passenger behavior at a large scale, saving expensive data collection efforts. For a review of the spectrum of applications – from strategic demand estimation to operational service performance measurements. The FairAccess project leverages on the availability of Access-kort data for the vast majority of trips performed in Stockholm County. The overarching goal of this project is to develop means to analyse empirically the impacts of policy/planning measures based on disaggregate passively collected smart card data. This involves a series of analysis and modelling challenges. We develop and apply a series algorithms to infer of tap-out locations, infer vehicles and travel times, and infer transfers to that journeys can be composed. Tap-in records have been matched with corresponding inferred tap-out locations and time stamps for about 80% of all records. Thereafter, we construct time-dependent origin-destination matrices for which segmentations can be performed with respect to geographical and user product features. We demonstrate the approach and algorithms developed by performing a before-after analysis of the fare scheme change from zone-based to flat fares. We analyse changes in travel patterns and derive price elasticities for distinctive market segments. The introduced fare policy delivered the desirable result of an increased ridership through improved convenience of the single-use products. Nevertheless, the significance of the service convenience component was underestimated, which resulted in the price adjustments being not in line with the mobility effects. The planning and development of the Stockholm public transport system must rely on the best empirical foundations available to support evidence-based decision-making and make the right priorities. To this end, the development and analysis performed in the FairAccess project lay a necessary foundation for further methodological developments and analyses such as on-board crowding evaluation, demand forecasting and identifying user groups.QC 20230726</p

How fair is the fare? Estimating travel patterns and the impacts of fare schemes for different user groups in Stockholm based on smartcard data : Final report for Trafik och Region 2018 SLL-KTH research project

There is a rapid increase in the deployment, acquisition and analysis of automated fare collection (AFC) systems, enabling a profound change in the ability to analyze high-volume data that relate to observed passenger travel behavior and recurrent patterns. The analysis of such passively collected data offers direct access to a continuous flow of observed passenger behavior at a large scale, saving expensive data collection efforts. For a review of the spectrum of applications – from strategic demand estimation to operational service performance measurements. The FairAccess project leverages on the availability of Access-kort data for the vast majority of trips performed in Stockholm County. The overarching goal of this project is to develop means to analyse empirically the impacts of policy/planning measures based on disaggregate passively collected smart card data. This involves a series of analysis and modelling challenges. We develop and apply a series algorithms to infer of tap-out locations, infer vehicles and travel times, and infer transfers to that journeys can be composed. Tap-in records have been matched with corresponding inferred tap-out locations and time stamps for about 80% of all records. Thereafter, we construct time-dependent origin-destination matrices for which segmentations can be performed with respect to geographical and user product features. We demonstrate the approach and algorithms developed by performing a before-after analysis of the fare scheme change from zone-based to flat fares. We analyse changes in travel patterns and derive price elasticities for distinctive market segments. The introduced fare policy delivered the desirable result of an increased ridership through improved convenience of the single-use products. Nevertheless, the significance of the service convenience component was underestimated, which resulted in the price adjustments being not in line with the mobility effects. The planning and development of the Stockholm public transport system must rely on the best empirical foundations available to support evidence-based decision-making and make the right priorities. To this end, the development and analysis performed in the FairAccess project lay a necessary foundation for further methodological developments and analyses such as on-board crowding evaluation, demand forecasting and identifying user groups.QC 20230726</p

Public transport fare elasticities from smartcard data: Evidence from a natural experiment

This paper develops a method for analysing the elasticity of travel demand to public transport fares. The methodology utilizes public transport smartcard data for collecting disaggregate full population data about passengers’ travel behaviour. The study extends previous work by deriving specific fare elasticities for distinct socioeconomic (e.g., car ownership and income) groups and public transport modes (metro, trains and buses), and by considering the directionality of the fare change. The case study involves a public transport fare policy introduced by the regional administration of Stockholm County in January 2017, where the zonal fare system for single-trip tickets was replaced by a flat-fare policy. The overall fare elasticity of travel funds is found to be −0.46. User sensitivity grows along with the journey distance. Metro users demonstrate the lowest sensitivity, followed by bus and commuter train riders. Low socioeconomic groups, in particular with respect to car ownership, tend to be less sensitive than the high-factor groups. In addition to the direct effect of changed fares, simplification and unification of the fare scheme appears to have substantially contributed to its attractiveness. The flat fare may allow the geographic disparity of public transport travel to be reduced and new users to be attracted from remote areas who are more prone to own cars