Constant Sequence Extension for Fast Search Using Weighted Hamming Distance

Representing visual data using compact binary codes is attracting increasing attention as binary codes are used as direct indices into hash table(s) for fast non-exhaustive search. Recent methods show that ranking binary codes using weighted Hamming distance (WHD) rather than Hamming distance (HD) by generating query-adaptive weights for each bit can better retrieve query-related items. However, search using WHD is slower than that using HD. One main challenge is that the complexity of extending a monotone increasing sequence using WHD to probe buckets in hash table(s) for existing methods is at least proportional to the square of the sequence length, while that using HD is proportional to the sequence length. To overcome this challenge, we propose a novel fast non-exhaustive search method using WHD. The key idea is to design a constant sequence extension algorithm to perform each sequence extension in constant computational complexity and the total complexity is proportional to the sequence length, which is justified by theoretical analysis. Experimental results show that our method is faster than other WHD-based search methods. Also, compared with the HD-based non-exhaustive search method, our method has comparable efficiency but retrieves more query-related items for the dataset of up to one billion items

ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection

Class-incremental learning (CIL) learns a classification model with training data of different classes arising progressively. Existing CIL either suffers from serious accuracy loss due to catastrophic forgetting, or invades data privacy by revisiting used exemplars. Inspired by linear learning formulations, we propose an analytic class-incremental learning (ACIL) with absolute memorization of past knowledge while avoiding breaching of data privacy (i.e., without storing historical data). The absolute memorization is demonstrated in the sense that class-incremental learning using ACIL given present data would give identical results to that from its joint-learning counterpart which consumes both present and historical samples. This equality is theoretically validated. Data privacy is ensured since no historical data are involved during the learning process. Empirical validations demonstrate ACIL's competitive accuracy performance with near-identical results for various incremental task settings (e.g., 5-50 phases). This also allows ACIL to outperform the state-of-the-art methods for large-phase scenarios (e.g., 25 and 50 phases).Comment: published in NeurIPS 202

Mitigating Memorization of Noisy Labels by Clipping the Model Prediction

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.Comment: Accepted by ICML 202

Speeding up deep neural network training with decoupled and analytic learning

Training deep neural networks usually demands a significantly long period of time. In this thesis, we explore methods in two different areas, i.e., decoupled learning and analytic learning, in order to reduce the training time. In decoupled learning, new methods are proposed to alleviate the sequential nature of the backpropagation (BP) which accounts for the most common means of training deep neural networks. The BP requires a sequential passing of activations and gradients, which has been recognized as lockings (i.e., the forward, backward, and update lockings). These lockings impose strong synchronism among modules (a consecutive stack of layers), rendering most modules idle during training. A fully decoupled learning method using delayed gradients (FDG) is first proposed which addresses all the three lockings. The FDG improves training efficiency as a significant acceleration is achieved. Furthermore, the decoupled learning inevitably introduces asynchronism that causes gradient staleness (also known as stale gradient effect), resulting in degraded generalization performance or even divergence. An accumulated decoupled learning (ADL) is hence developed to cope with the staleness issue. The proposed ADL is proved to be effective in reducing the gradient staleness both theoretically and empirically, demonstrating an improved generalization ability compared with that of the current works which ignore the staleness. New methods are also developed in the area of analytic learning by discarding the BP entirely and training the network using analytical solutions. The analytic learning trains neural networks in an exceedingly fast fashion as the training is completed within one single epoch. There are two main challenges in this area. The first challenge lies in the difficulty of finding analytical solutions for multilayer networks. Existing methods have several limitations, such as structural constraints or requesting invertible activation functions. Here a correlation projection network (CPNet) is developed which removes the aforementioned limitations by treating the network as a combination of multiple 2-layer modules. The analytic learning of CPNet is made possible after the label information is projected into the hidden modules so that each 2-layer module can analytically solve the locally supervised learning using the least squares solutions. The other challenge is that, to implement the analytic learning, there is a possible issue of memory leak caused by matrix operations based on the entire dataset. Hence, a block-wise recursive Moore-Penrose inverse (BRMP) method is proposed which can reformulate the original analytic learning exactly into a block-wise alternative using a block-wise decomposition of Moore-Penrose inverse. The BRMP not only reduces the memory consumption while keeping its high training efficiency, but also takes care of the potential rank-deficient matrix inversion issue during the analytic learning.Doctor of Philosoph

Augmented EMD for complex-valued univariate signals

In this study, the authors propose an efficient extension of the standard empirical mode decomposition (EMD) for complex-valued univariate signal decomposition. The key idea of the extension is to convert a complex-valued univariate signal into a longer real-valued signal by augmenting the real part with the flipped imaginary part, and then to decompose it into intrinsic mode functions (IMFs) using the EMD once only. The bivariate IMFs are then retrieved from the obtained IMFs. Their empirical results on synthetic data show that the proposed method significantly outperforms the traditional bivariate EMD (BEMD) method in terms of computational efficiency while producing a comparable extraction error. Moreover, the proposed method shows better micro-Doppler signature analysis performance on physically measured continuous-wave radar data than that of the BEMD.Accepted versio

Machine Learning Techniques and Systems for Mask-Face Detection—Survey and a New OOD-Mask Approach

Mask-face detection has been a significant task since the outbreak of the COVID-19 pandemic in early 2020. While various reviews on mask-face detection techniques up to 2021 are available, little has been reviewed on the distinction between two-class (i.e., wearing mask and without mask) and three-class masking, which includes an additional incorrect-mask-wearing class. Moreover, no formal review has been conducted on the techniques of implementing mask detection models in hardware systems or mobile devices. The objectives of this paper are three-fold. First, we aimed to provide an up-to-date review of recent mask-face detection research in both two-class cases and three-class cases, next, to fill the gap left by existing reviews by providing a formal review of mask-face detection hardware systems; and to propose a new framework named Out-of-distribution Mask (OOD-Mask) to perform the three-class detection task using only two-class training data. This was achieved by treating the incorrect-mask-wearing scenario as an anomaly, leading to reasonable performance in the absence of training data of the third class

Machine learning techniques and systems for mask-face detection—survey and a new OOD-mask approach

Mask-face detection has been a significant task since the outbreak of the COVID-19 pandemic in early 2020. While various reviews on mask-face detection techniques up to 2021 are available, little has been reviewed on the distinction between two-class (i.e., wearing mask and without mask) and three-class masking, which includes an additional incorrect-mask-wearing class. Moreover, no formal review has been conducted on the techniques of implementing mask detection models in hardware systems or mobile devices. The objectives of this paper are three-fold. First, we aimed to provide an up-to-date review of recent mask-face detection research in both two-class cases and three-class cases, next, to fill the gap left by existing reviews by providing a formal review of mask-face detection hardware systems; and to propose a new framework named Out-of-distribution Mask (OOD-Mask) to perform the three-class detection task using only two-class training data. This was achieved by treating the incorrect-mask-wearing scenario as an anomaly, leading to reasonable performance in the absence of training data of the third class.Agency for Science, Technology and Research (A*STAR)Published versionThis work was supported by the Science and Engineering Research Council, Agency of Science, Technology and Research, Singapore, through the National Robotics Program under Grant No. 1922500054

Energy-Efficient Virtual Network Embedding Algorithm Based on Hopfield Neural Network

To solve the energy-efficient virtual network embedding problem, this study proposes an embedding algorithm based on Hopfield neural network. An energy-efficient virtual network embedding model was established. Wavelet diffusion was performed to take the structural feature value into consideration and provide a candidate set for virtual network embedding. In addition, the Hopfield network was used in the candidate set to solve the virtual network energy-efficient embedding problem. The augmented Lagrangian multiplier method was used to transform the energy-efficient virtual network embedding constraint problem into an unconstrained problem. The resulting unconstrained problem was used as the energy function of the Hopfield network, and the network weight was iteratively trained. The energy-efficient virtual network embedding scheme was obtained when the energy function was balanced. To prove the effectiveness of the proposed algorithm, we designed two experimental environments, namely, a medium-sized scenario and a small-sized scenario. Simulation results show that the proposed algorithm achieved a superior performance and effectively decreased the energy consumption relative to the other methods in both scenarios. Furthermore, the proposed algorithm reduced the number of open nodes and open links leading to a reduction in the overall power consumption of the virtual network embedding process, while ensuring the average acceptance ratio and the average ratio of the revenue and cost

Acceptance of Online Mapping Technology among Older Adults: Technology Acceptance Model with Facilitating Condition, Compatibility, and Self-Satisfaction

The benefits of traveling for older adults are extensively supported in the literature. Online mapping technology (OMT) is one of the most widely used applications by people during traveling. This study aimed to obtain insight into the acceptance of OMT among older adults. Additionally, an OMT acceptance model for older adults was developed in this study by integrating facilitating condition (FC), compatibility (COM), and self-satisfaction (SS) into the technology acceptance model (TAM). In this study, structural equation modeling was applied to the test of the OMT acceptance model. This study adopted a cross-sectional structured questionnaire survey for collecting quantitative data from older adults in China. Four hundred and sixteen Chinese older adults were involved in this survey. This study found that TAM was useful to explain the OMT acceptance among older adults. Additionally, FC was confirmed to be a positive factor in determining the perceived ease of use, while COM and SS were found to positively influence perceived usefulness. The results of this study are helpful for OMT developers to design OMT and adopt measures to enhance the use of OMT among older adults, thereby increasing their travel frequency