A Cost-based Optimizer for Gradient Descent Optimization

As the use of machine learning (ML) permeates into diverse application domains, there is an urgent need to support a declarative framework for ML. Ideally, a user will specify an ML task in a high-level and easy-to-use language and the framework will invoke the appropriate algorithms and system configurations to execute it. An important observation towards designing such a framework is that many ML tasks can be expressed as mathematical optimization problems, which take a specific form. Furthermore, these optimization problems can be efficiently solved using variations of the gradient descent (GD) algorithm. Thus, to decouple a user specification of an ML task from its execution, a key component is a GD optimizer. We propose a cost-based GD optimizer that selects the best GD plan for a given ML task. To build our optimizer, we introduce a set of abstract operators for expressing GD algorithms and propose a novel approach to estimate the number of iterations a GD algorithm requires to converge. Extensive experiments on real and synthetic datasets show that our optimizer not only chooses the best GD plan but also allows for optimizations that achieve orders of magnitude performance speed-up.Comment: Accepted at SIGMOD 201

Insulator-to-Metal Transition in Selenium-Hyperdoped Silicon: Observation and Origin

Hyperdoping has emerged as a promising method for designing semiconductors with unique optical and electronic properties, although such properties currently lack a clear microscopic explanation. Combining computational and experimental evidence, we probe the origin of sub-band gap optical absorption and metallicity in Se-hyperdoped Si. We show that sub-band gap absorption arises from direct defect-to-conduction band transitions rather than free carrier absorption. Density functional theory predicts the Se-induced insulator-to-metal transition arises from merging of defect and conduction bands, at a concentration in excellent agreement with experiment. Quantum Monte Carlo calculations confirm the critical concentration, demonstrate that correlation is important to describing the transition accurately, and suggest that it is a classic impurity-driven Mott transition.Comment: 5 pages, 3 figures (PRL formatted

DeepWalk: Online Learning of Social Representations

We present DeepWalk, a novel approach for learning latent representations of vertices in a network. These latent representations encode social relations in a continuous vector space, which is easily exploited by statistical models. DeepWalk generalizes recent advancements in language modeling and unsupervised feature learning (or deep learning) from sequences of words to graphs. DeepWalk uses local information obtained from truncated random walks to learn latent representations by treating walks as the equivalent of sentences. We demonstrate DeepWalk's latent representations on several multi-label network classification tasks for social networks such as BlogCatalog, Flickr, and YouTube. Our results show that DeepWalk outperforms challenging baselines which are allowed a global view of the network, especially in the presence of missing information. DeepWalk's representations can provide $F_1$ scores up to 10% higher than competing methods when labeled data is sparse. In some experiments, DeepWalk's representations are able to outperform all baseline methods while using 60% less training data. DeepWalk is also scalable. It is an online learning algorithm which builds useful incremental results, and is trivially parallelizable. These qualities make it suitable for a broad class of real world applications such as network classification, and anomaly detection.Comment: 10 pages, 5 figures, 4 table

Asynchronous Training of Word Embeddings for Large Text Corpora

Word embeddings are a powerful approach for analyzing language and have been widely popular in numerous tasks in information retrieval and text mining. Training embeddings over huge corpora is computationally expensive because the input is typically sequentially processed and parameters are synchronously updated. Distributed architectures for asynchronous training that have been proposed either focus on scaling vocabulary sizes and dimensionality or suffer from expensive synchronization latencies. In this paper, we propose a scalable approach to train word embeddings by partitioning the input space instead in order to scale to massive text corpora while not sacrificing the performance of the embeddings. Our training procedure does not involve any parameter synchronization except a final sub-model merge phase that typically executes in a few minutes. Our distributed training scales seamlessly to large corpus sizes and we get comparable and sometimes even up to 45% performance improvement in a variety of NLP benchmarks using models trained by our distributed procedure which requires $1/10$ of the time taken by the baseline approach. Finally we also show that we are robust to missing words in sub-models and are able to effectively reconstruct word representations.Comment: This paper contains 9 pages and has been accepted in the WSDM201

Quantitative analysis of the loss of distinction between gray and white matter in comatose patients after cardiac arrest

BACKGROUND AND PURPOSE: Anecdotal reports suggest that a loss of distinction between gray (GM) and white matter (WM) as adjudged by CT scan predicts poor outcome in comatose patients after cardiac arrest. To address this, we quantitatively assessed GM and WM intensities at various brain levels in comatose patients after cardiac arrest. METHODS: Patients for whom consultation was requested within 24 hours of a cardiac arrest were identified with the use of a computerized database that tracks neurological consultations at our institution. Twenty-five comatose patients were identified for whom complete medical records and CT scans were available for review. Twenty-five consecutive patients for whom a CT scan was interpreted as normal served as controls. Hounsfield units (HUs) were measured in small defined areas obtained from axial images at the levels of the basal ganglia, centrum semiovale, and high convexity area. RESULTS: At each level tested, lower GM intensity and higher WM intensity were noted in comatose patients compared with normal controls. The GM/WM ratio was significantly lower among comatose patients compared with controls (P:\u3c0.0001, rank sum test). There was essentially no overlap in GM/WM ratios between control and study patients. The difference was greatest at the basal ganglia level. We also observed a marginally significant difference in the GM/WM ratio at the basal ganglia level between those patients who died and those who survived cardiac arrest (P:=0. 035, 1-tailed t test). Using receiver operating characteristic curve analysis, we determined that a difference in GM/WM ratio of \u3c1.18 at the basal ganglia level was 100% predictive of death. At the basal ganglia level, none of 12 patients below this threshold survived, whereas the survival rate was 46% among patients in whom the ratio was \u3e1.18. The empirical risk of death was 21.67 for comatose patients with a value below threshold. CONCLUSIONS: The ratio in HUs of GM to WM provides a reproducible measure of the distinction between gray and white matter. A lower GM/WM ratio is observed in comatose patients immediately after cardiac arrest. The basal ganglia level seems to be the most sensitive location on CT for measuring this relationship. Although a GM/WM ratio \u3c1.18 at this level predicted death in this retrospective study, the difference in this study is not robust enough to recommend that management decisions be dictated by CT results. The results, however, do warrant consideration of a prospective study to determine the reliability of CT scanning in predicting outcome for comatose patients after cardiac arrest

Insulator-to-metal transition in sulfur-doped silicon

We observe an insulator-to-metal (I-M) transition in crystalline silicon doped with sulfur to non- equilibrium concentrations using ion implantation followed by pulsed laser melting and rapid resolidification. This I-M transition is due to a dopant known to produce only deep levels at equilibrium concentrations. Temperature-dependent conductivity and Hall effect measurements for temperatures T > 1.7 K both indicate that a transition from insulating to metallic conduction occurs at a sulfur concentration between 1.8 and 4.3 x 10^20 cm-3. Conduction in insulating samples is consistent with variable range hopping with a Coulomb gap. The capacity for deep states to effect metallic conduction by delocalization is the only known route to bulk intermediate band photovoltaics in silicon.Comment: Submission formatting; 4 journal pages equivalen

Methodology for vetting heavily doped semiconductors for intermediate band photovoltaics: A case study in sulfur-hyperdoped silicon

We present a methodology for estimating the efficiency potential for candidate impurity-band photovoltaic materials from empirical measurements. This methodology employs both Fourier transform infrared spectroscopy and low-temperature photoconductivity to calculate a “performance figure of merit” and to determine both the position and bandwidth of the impurity band. We evaluate a candidate impurity-band material, silicon hyperdoped with sulfur; we find that the figure of merit is more than one order of magnitude too low for photovoltaic devices that exceed the thermodynamic efficiency limit for single band gap materials.National Science Foundation (U.S.) (Energy, Power, and Adaptive Systems Grant Contract ECCS-1102050)National Science Foundation (U.S.) (United States. Dept. of Energy NSF CA EEC-1041895)Center for Clean Water and Clean Energy at MIT and KFUP

Tensor completion in hierarchical tensor representations

Compressed sensing extends from the recovery of sparse vectors from undersampled measurements via efficient algorithms to the recovery of matrices of low rank from incomplete information. Here we consider a further extension to the reconstruction of tensors of low multi-linear rank in recently introduced hierarchical tensor formats from a small number of measurements. Hierarchical tensors are a flexible generalization of the well-known Tucker representation, which have the advantage that the number of degrees of freedom of a low rank tensor does not scale exponentially with the order of the tensor. While corresponding tensor decompositions can be computed efficiently via successive applications of (matrix) singular value decompositions, some important properties of the singular value decomposition do not extend from the matrix to the tensor case. This results in major computational and theoretical difficulties in designing and analyzing algorithms for low rank tensor recovery. For instance, a canonical analogue of the tensor nuclear norm is NP-hard to compute in general, which is in stark contrast to the matrix case. In this book chapter we consider versions of iterative hard thresholding schemes adapted to hierarchical tensor formats. A variant builds on methods from Riemannian optimization and uses a retraction mapping from the tangent space of the manifold of low rank tensors back to this manifold. We provide first partial convergence results based on a tensor version of the restricted isometry property (TRIP) of the measurement map. Moreover, an estimate of the number of measurements is provided that ensures the TRIP of a given tensor rank with high probability for Gaussian measurement maps.Comment: revised version, to be published in Compressed Sensing and Its Applications (edited by H. Boche, R. Calderbank, G. Kutyniok, J. Vybiral

Supersaturating silicon with transition metals by ion implantation and pulsed laser melting

We investigate the possibility of creating an intermediate band semiconductor by supersaturating Si with a range of transition metals (Au, Co, Cr, Cu, Fe, Pd, Pt, W, and Zn) using ion implantation followed by pulsed laser melting (PLM). Structural characterization shows evidence of either surface segregation or cellular breakdown in all transition metals investigated, preventing the formation of high supersaturations. However, concentration-depth profiling reveals that regions of Si supersaturated with Au and Zn are formed below the regions of cellular breakdown. Fits to the concentration-depth profile are used to estimate the diffusive speeds, v [subscript D], of Au and Zn, and put lower bounds on v [subscript D] of the other metals ranging from 10[superscript 2] to 10[superscript 4] m/s. Knowledge of v [subscript D] is used to tailor the irradiation conditions and synthesize single-crystal Si supersaturated with 10[superscript 19] Au/cm[superscript 3] without cellular breakdown. Values of v [subscript D] are compared to those for other elements in Si. Two independent thermophysical properties, the solute diffusivity at the melting temperature, D [subscript s](T [subscript m]), and the equilibrium partition coefficient, k [subscript e], are shown to simultaneously affect v [subscript D]. We demonstrate a correlation between v [subscript D] and the ratio D [subscript s](T [subscript m])/k [subscript e] [superscript 0.67], which is exhibited for Group III, IV, and V solutes but not for the transition metals investigated. Nevertheless, comparison with experimental results suggests that D [subscript s](T [subscript m])/k [subscript e] [superscript 0.67] might serve as a metric for evaluating the potential to supersaturate Si with transition metals by PLM.National Science Foundation (U.S.) (Faculty Early Career Development Program ECCS-1150878)Chesonis Family FoundationUnited States. Army Research Laboratory (United States. Army Research Office Grant W911NF-10-1-0442)National Science Foundation (U.S.) (United States. Dept. of Energy NSF CA EEC-1041895

Supersaturating silicon with transition metals by ion implantation and pulsed laser melting

We investigate the possibility of creating an intermediate band semiconductor by supersaturating Si with a range of transition metals (Au, Co, Cr, Cu, Fe, Pd, Pt, W, and Zn) using ion implantation followed by pulsed laser melting (PLM). Structural characterization shows evidence of either surface segregation or cellular breakdown in all transition metals investigated, preventing the formation of high supersaturations. However, concentration-depth profiling reveals that regions of Si supersaturated with Au and Zn are formed below the regions of cellular breakdown. Fits to the concentration-depth profile are used to estimate the diffusive speeds, v D, of Au and Zn, and put lower bounds on v D of the other metals ranging from 10² to 10⁴ m/s. Knowledge of v D is used to tailor the irradiation conditions and synthesize single-crystal Si supersaturated with 10¹⁹ Au/cm³ without cellular breakdown. Values of v D are compared to those for other elements in Si. Two independent thermophysical properties, the solute diffusivity at the melting temperature, D s(T m), and the equilibrium partition coefficient, k e, are shown to simultaneously affect v D. We demonstrate a correlation between v D and the ratio D s(T m)/k e ⁰·⁶⁷, which is exhibited for Group III, IV, and V solutes but not for the transition metals investigated. Nevertheless, comparison with experimental results suggests that D s(T m)/k e ⁰·⁶⁷ might serve as a metric for evaluating the potential to supersaturate Si with transition metals by PLM.Research at Harvard was supported by The U.S. Army Research Office under contracts W911NF-12-1-0196 and W911NF-09-1-0118. M.T.W. and T.B.’s work was supported by the U.S. Army Research Laboratory and the U.S. Army Research Office under Grant No. W911NF-10-1-0442, and the National Science Foundation (NSF) Faculty Early Career Development Program ECCS-1150878 (to T.B.). M.J.S., J.T.S., M.T.W., T.B., and S.G. acknowledge a generous gift from the Chesonis Family Foundation and support in part by the National Science Foundation (NSF) and the Department of Energy (DOE) under NSF CA No. EEC- 1041895. S.C. and J.S.W.’s work was supported by The Australian Research Council. J.M. was supported by a National Research Council Research Associateship