CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra

Many areas of machine learning and science involve large linear algebra problems, such as eigendecompositions, solving linear systems, computing matrix exponentials, and trace estimation. The matrices involved often have Kronecker, convolutional, block diagonal, sum, or product structure. In this paper, we propose a simple but general framework for large-scale linear algebra problems in machine learning, named CoLA (Compositional Linear Algebra). By combining a linear operator abstraction with compositional dispatch rules, CoLA automatically constructs memory and runtime efficient numerical algorithms. Moreover, CoLA provides memory efficient automatic differentiation, low precision computation, and GPU acceleration in both JAX and PyTorch, while also accommodating new objects, operations, and rules in downstream packages via multiple dispatch. CoLA can accelerate many algebraic operations, while making it easy to prototype matrix structures and algorithms, providing an appealing drop-in tool for virtually any computational effort that requires linear algebra. We showcase its efficacy across a broad range of applications, including partial differential equations, Gaussian processes, equivariant model construction, and unsupervised learning.Comment: Code available at https://github.com/wilson-labs/col

A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks

Unlike conventional grid and mesh based methods for solving partial differential equations (PDEs), neural networks have the potential to break the curse of dimensionality, providing approximate solutions to problems where using classical solvers is difficult or impossible. While global minimization of the PDE residual over the network parameters works well for boundary value problems, catastrophic forgetting impairs the applicability of this approach to initial value problems (IVPs). In an alternative local-in-time approach, the optimization problem can be converted into an ordinary differential equation (ODE) on the network parameters and the solution propagated forward in time; however, we demonstrate that current methods based on this approach suffer from two key issues. First, following the ODE produces an uncontrolled growth in the conditioning of the problem, ultimately leading to unacceptably large numerical errors. Second, as the ODE methods scale cubically with the number of model parameters, they are restricted to small neural networks, significantly limiting their ability to represent intricate PDE initial conditions and solutions. Building on these insights, we develop Neural IVP, an ODE based IVP solver which prevents the network from getting ill-conditioned and runs in time linear in the number of parameters, enabling us to evolve the dynamics of challenging PDEs with neural networks.Comment: ICLR 2023. Code available at https://github.com/mfinzi/neural-iv

PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization

While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works. In this paper, we develop a compression approach based on quantizing neural network parameters in a linear subspace, profoundly improving on previous results to provide state-of-the-art generalization bounds on a variety of tasks, including transfer learning. We use these tight bounds to better understand the role of model size, equivariance, and the implicit biases of optimization, for generalization in deep learning. Notably, we find large models can be compressed to a much greater extent than previously known, encapsulating Occam's razor. We also argue for data-independent bounds in explaining generalization.Comment: NeurIPS 2022. Code is available at https://github.com/activatedgeek/tight-pac-baye

Identification of Interdependent Variables that Influence Coreceptor Switch in R5 SHIVSF162P3N_{SF162P3N}-Infected Macaques

Background: We previously reported that adoption of an “open” envelope glycoprotein (Env) to expose the CD4 binding site for efficient receptor binding and infection of cell targets such as macrophages that express low levels of the receptor represents an early event in the process of coreceptor switch in two rapidly progressing (RP) R5 SHIV$_{SF162P3N}$-infected rhesus macaques, releasing or reducing Env structural constraints that have been suggested to limit the pathways available for a change in coreceptor preference. Here we extended these studies to two additional RP monkeys with coreceptor switch and three without to confirm and identify additional factors that facilitated the process of phenotypic conversion. Results: We found that regardless of coreceptor switching, R5 viruses in SHIV$_{SF162P3N}$-infected RP macaques evolved over time to infect macrophages more efficiently; this was accompanied by increased sCD4 sensitivity, with structural changes in the CD4 binding site, the V3 loop and/or the fusion domain of their Envs that are suggestive of better CD4 contact, CCR5 usage and/or virus fusion. However, sCD4-sensitive variants with improved CD4 binding were observed only in RPs with coreceptor switch. Furthermore, cumulative viral load was higher in RPs with than in those without phenotypic switch, with the latter maintaining a longer period of seroconversion. Conclusions: Our data suggest that the increased virus replication in the RPs with R5-to-X4 conversion increased the rate of virus evolution and reduction in the availability of target cells with optimal CD4 expression heightened the competition for binding to the receptor. In the absence of immunological restrictions, variants that adopt an “open” Env to expose the CD4 binding site for better CD4 use are selected, allowing structural changes that confer CXCR4-use to be manifested. Viral load, change in target cell population during the course of infection and host immune response therefore are interdependent variables that influence R5 virus evolution and coreceptor switch in SHIV$_{SF162P3N}$-infected rhesus macaques. Because an "open" Env conformation also renders the virus more susceptible to antibody neutralization, our findings help to explain the infrequent and late appearance of X4 virus in HIV-1 infection when the immune system deteriorates

An Inducible Cell-Cell Fusion System with Integrated Ability to Measure the Efficiency and Specificity of HIV-1 Entry Inhibitors

HIV-1 envelope glycoproteins (Envs) mediate virus entry by fusing the viral and target cell membranes, a multi-step process that represents an attractive target for inhibition. Entry inhibitors with broad-range activity against diverse isolates of HIV-1 may be extremely useful as lead compounds for the development of therapies or prophylactic microbicides. To facilitate the identification of such inhibitors, we have constructed a cell-cell fusion system capable of simultaneously monitoring inhibition efficiency and specificity. In this system, effector cells stably express a tetracycline-controlled transactivator (tTA) that enables tightly inducible expression of both HIV-1 Env and the Renilla luciferase (R-Luc) reporter protein. Target cells express the HIV-1 receptors, CD4 and CCR5, and carry the firefly luciferase (F-Luc) reporter gene under the control of a tTA-responsive promoter. Thus, Env-mediated fusion of these two cell types allows the tTA to diffuse to the target cell and activate the expression of the F-Luc protein. The efficiency with which an inhibitor blocks cell-cell fusion is measured by a decrease in the F-Luc activity, while the specificity of the inhibitor is evaluated by its effect on the R-Luc activity. The system exhibited a high dynamic range and high Z'-factor values. The assay was validated with a reference panel of inhibitors that target different steps in HIV-1 entry, yielding inhibitory concentrations comparable to published virus inhibition data. Our system is suitable for large-scale screening of chemical libraries and can also be used for detailed characterization of inhibitory and cytotoxic properties of known entry inhibitors

Results from a Large, Multinational Sample Using the Childhood Trauma Questionnaire

Childhood maltreatment has diverse, lifelong impact on morbidity and mortality. The Childhood Trauma Questionnaire (CTQ) is one of the most commonly used scales to assess and quantify these experiences and their impact. Curiously, despite very widespread use of the CTQ, scores on its Minimization-Denial (MD) subscale—originally designed to assess a positive response bias—are rarely reported. Hence, little is known about this measure. If response biases are either common or consequential, current practices of ignoring the MD scale deserve revision. Therewith, we designed a study to investigate 3 aspects of minimization, as defined by the CTQ’s MD scale: 1) its prevalence; 2) its latent structure; and finally 3) whether minimization moderates the CTQ’s discriminative validity in terms of distinguishing between psychiatric patients and community volunteers. Archival, item-level CTQ data from 24 multinational samples were combined for a total of 19,652 participants. Analyses indicated: 1) minimization is common; 2) minimization functions as a continuous construct; and 3) high MD scores attenuate the ability of the CTQ to distinguish between psychiatric patients and community volunteers. Overall, results suggest that a minimizing response bias—as detected by the MD subscale—has a small but significant moderating effect on the CTQ’s discriminative validity. Results also may suggest that some prior analyses of maltreatment rates or the effects of early maltreatment that have used the CTQ may have underestimated its incidence and impact. We caution researchers and clinicians about the widespread practice of using the CTQ without the MD or collecting MD data but failing to assess and control for its effects on outcomes or dependent variables

Adoption of an “Open” Envelope Conformation Facilitating CD4 Binding and Structural Remodeling Precedes Coreceptor Switch in R5 SHIV-Infected Macaques

A change in coreceptor preference from CCR5 to CXCR4 towards the end stage disease in some HIV-1 infected individuals has been well documented, but the reasons and mechanisms for this tropism switch remain elusive. It has been suggested that envelope structural constraints in accommodating amino acid changes required for CXCR4 usage is an obstacle to tropism switch, limiting the rate and pathways available for HIV-1 coreceptor switching. The present study was initiated in two R5 SHIVSF162P3N-infected rapid progressor macaques with coreceptor switch to test the hypothesis that an early step in the evolution of tropism switch is the adoption of a less constrained and more “open” envelope conformation for better CD4 usage, allowing greater structural flexibility to accommodate further mutational changes that confer CXCR4 utilization. We show that, prior to the time of coreceptor switch, R5 viruses in both macaques evolved to become increasingly sCD4-sensitive, suggestive of enhanced exposure of the CD4 binding site and an “open” envelope conformation, and this correlated with better gp120 binding to CD4 and with more efficient infection of CD4low cells such as primary macrophages. Moreover, significant changes in neutralization sensitivity to agents and antibodies directed against functional domains of gp120 and gp41 were seen for R5 viruses close to the time of X4 emergence, consistent with global changes in envelope configuration and structural plasticity. These observations in a simian model of R5-to-X4 evolution provide a mechanistic basis for the HIV-1 coreceptor switch

Natural HIV-1 Nef Polymorphisms Impair SERINC5 Downregulation Activity

HIV-1 Nef enhances virion infectivity by counteracting host restriction factor SERINC5; however, the impact of natural Nef polymorphisms on this function is largely unknown. We characterize SERINC5 downregulation activity of 91 primary HIV-1 subtype B nef alleles, including isolates from 45 elite controllers and 46 chronic progressors. Controller-derived Nef clones display lower ability to downregulate SERINC5 (median 80% activity) compared with progressor-derived clones (median 96% activity) (p = 0.0005). We identify 18 Nef polymorphisms associated with differential function, including two CTL escape mutations that contribute to lower SERINC5 downregulation: K94E, driven by HLA-B( *)08, and H116N, driven by the protective allele HLA-B( *)57. HIV-1 strains encoding Nef K94E and/or H116N display lower infectivity and replication capacity in the presence of SERINC5. Our results demonstrate that natural polymorphisms in HIV-1 Nef can impair its ability to internalize SERINC5, indicating that variation in this recently described function may contribute to differences in viral pathogenesis