FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. However, because of the zero-hard rectification, ReLU networks miss the benefits from negative values. In this paper, we propose a novel activation function called \emph{flexible rectified linear unit (FReLU)} to further explore the effects of negative values. By redesigning the rectified point of ReLU as a learnable parameter, FReLU expands the states of the activation output. When the network is successfully trained, FReLU tends to converge to a negative value, which improves the expressiveness and thus the performance. Furthermore, FReLU is designed to be simple and effective without exponential functions to maintain low cost computation. For being able to easily used in various network architectures, FReLU does not rely on strict assumptions by self-adaption. We evaluate FReLU on three standard image classification datasets, including CIFAR-10, CIFAR-100, and ImageNet. Experimental results show that the proposed method achieves fast convergence and higher performances on both plain and residual networks

Energy Efficient Coordinated Beamforming for Multi-cell MISO Systems

In this paper, we investigate the optimal energy efficient coordinated beamforming in multi-cell multiple-input single-output (MISO) systems with $K$ multiple-antenna base stations (BS) and $K$ single-antenna mobile stations (MS), where each BS sends information to its own intended MS with cooperatively designed transmit beamforming. We assume single user detection at the MS by treating the interference as noise. By taking into account a realistic power model at the BS, we characterize the Pareto boundary of the achievable energy efficiency (EE) region of the $K$ links, where the EE of each link is defined as the achievable data rate at the MS divided by the total power consumption at the BS. Since the EE of each link is non-cancave (which is a non-concave function over an affine function), characterizing this boundary is difficult. To meet this challenge, we relate this multi-cell MISO system to cognitive radio (CR) MISO channels by applying the concept of interference temperature (IT), and accordingly transform the EE boundary characterization problem into a set of fractional concave programming problems. Then, we apply the fractional concave programming technique to solve these fractional concave problems, and correspondingly give a parametrization for the EE boundary in terms of IT levels. Based on this characterization, we further present a decentralized algorithm to implement the multi-cell coordinated beamforming, which is shown by simulations to achieve the EE Pareto boundary.Comment: 6 pages, 2 figures, to be presented in IEEE GLOBECOM 201

How to Fine-Tune BERT for Text Classification?

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets