Faster Boosting with Smaller Memory

State-of-the-art implementations of boosting, such as XGBoost and LightGBM, can process large training sets extremely fast. However, this performance requires that the memory size is sufficient to hold a 2-3 multiple of the training set size. This paper presents an alternative approach to implementing the boosted trees, which achieves a significant speedup over XGBoost and LightGBM, especially when the memory size is small. This is achieved using a combination of three techniques: early stopping, effective sample size, and stratified sampling. Our experiments demonstrate a 10-100 speedup over XGBoost when the training data is too large to fit in memory.Comment: NeurIPS 201

Experimental Design for Bathymetry Editing

We describe an application of machine learning to a real-world computer assisted labeling task. Our experimental results expose significant deviations from the IID assumption commonly used in machine learning. These results suggest that the common random split of all data into training and testing can often lead to poor performance.Comment: Published as a workshop paper at ICML 2020 Workshop on Real World Experiment Design and Active Learnin

Biomechanical investigation of the hybrid modified cortical bone screw–pedicle screw fixation technique: Finite-element analysis

BackgroundHybrid fixation techniques including the both modified cortical bone trajectory (MCBT) and traditional trajectory (TT) at the L4 and L5 lumbar segment are firstly proposed by our team. Therefore, the purpose of this study is to evaluate and provide specific biomechanical data of the hybrid fixation techniques including the MCBT and TT.MethodsFour human cadaveric specimens were from the anatomy laboratory of Xinjiang Medical University. Four finite-element (FE) models of the L4–L5 lumbar spine were generated. For each of them, four implanted models with the following fixations were established: TT-TT (TT screw at the cranial and caudal level), MCBT-MCBT (MCBT screw at the cranial and caudal level), hybrid MCBT-TT (MCBT screw at the cranial level and TT screw at the caudal level), and TT-MCBT (TT screw at the cranial level and MCBT screw at the caudal level). A 400-N compressive load with 7.5 N/m moments was applied to simulate flexion, extension, lateral bending, and rotation, respectively. The range of motion (ROM) of the L4–L5 segment and the posterior fixation, the von Mises stress of the intervertebral disc, and the posterior fixation were compared.ResultsCompared to the TT-TT group, the MCBT-TT showed a significant lower ROM of the L4–L5 segment (p ≤ 0.009), lower ROM of the posterior fixation (p < 0.001), lower intervertebral disc stress (p < 0.001), and lower posterior fixation stress (p ≤ 0.041). TT-MCBT groups showed a significant lower ROM of the L4–L5 segment (p ≤ 0.012), lower ROM of the posterior fixation (p < 0.001), lower intervertebral disc stress (p < 0.001), and lower posterior fixation stress (p ≤ 0.038).ConclusionsThe biomechanical properties of the hybrid MCBT-TT and TT-MCBT techniques at the L4–L5 segment are superior to that of stability MCBT-MCBT and TT-TT techniques, and feasibility needs further cadaveric study to verify

Mean first-passage time for random walks on undirected networks

In this paper, by using two different techniques we derive an explicit formula for the mean first-passage time (MFPT) between any pair of nodes on a general undirected network, which is expressed in terms of eigenvalues and eigenvectors of an associated matrix similar to the transition matrix. We then apply the formula to derive a lower bound for the MFPT to arrive at a given node with the starting point chosen from the stationary distribution over the set of nodes. We show that for a correlated scale-free network of size $N$ with a degree distribution $P(d)\sim d^{-\gamma}$, the scaling of the lower bound is $N^{1-1/\gamma}$. Also, we provide a simple derivation for an eigentime identity. Our work leads to a comprehensive understanding of recent results about random walks on complex networks, especially on scale-free networks.Comment: 7 pages, no figures; definitive version published in European Physical Journal

Parallel Boosting and Learning from Diverse Datasets

This thesis is a study of boosting. It consists of two parts. In the first part, we develop a new way of parallelizing boosting. In the second part, we apply boosting to the problem of bathymetry data editing and study the issues of experimental design for diverse datasets.The first part of this thesis presents a parallel boosting algorithm that achieves a significant speedup while keeping a small memory footprint. It combines two novel techniques. One is a method for parallelization with weak synchronous requirement which we call "Tell Me Something New" (TMSN). The other is a method we call stratified weighted sampling that significantly reduces the I/O load of boosting.We implemented our algorithm using the Rust programming language and demonstrated its superior performance when memory size is limited. Our experiments show a 10-100x speedup over two of the popular implementations of boosted trees, XGBoost and LightGBM, when training data is too large to fit in memory.The second part of this thesis involves a project that uses boosting as an aid in the bathymetry data editing. Bathymetry is a study of the depths and shapes of underwater terrain. The objective of our project is to create a binary classifier that separates the correct depth measures from the incorrect ones. Our experimental results challenge the standard assumption that training and testing samples are both drawn i.i.d. from a fixed distribution.First, we examine spurious correlation, where some training and testing samples are similar to each other because they are duplicates, near-duplicates, or sequentially collected. A simple memorization-based model could achieve a low in-sample validation error in these cases, but its out-of-sample test error is much worse.Second, we examine data diversity, in which datasets are not diverse enough to be representative. It happens when the feature dimension is so high that collecting a representative sample is difficult. The models trained in these cases perform poorly on a new test set collected separately because of the domain shift problem.Lastly, we propose an alternative framework from the perspective of experimental design and present a case study with modeling bathymetry data editing

Experimental Design for Bathymetry Editing

We describe an application of machine learning to a real-world computer assisted labeling task. Our experimental results expose significant deviations from the IID assumption commonly used in machine learning. These results suggest that the common random split of all data into training and testing can often lead to poor performance