On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization

Extrapolation is a well-known technique for solving convex optimization and variational inequalities and recently attracts some attention for non-convex optimization. Several recent works have empirically shown its success in some machine learning tasks. However, it has not been analyzed for non-convex minimization and there still remains a gap between the theory and the practice. In this paper, we analyze gradient descent and stochastic gradient descent with extrapolation for finding an approximate first-order stationary point in smooth non-convex optimization problems. Our convergence upper bounds show that the algorithms with extrapolation can be accelerated than without extrapolation

Capacity Bounds for Broadcast Channels with Confidential Messages

In this paper, we study capacity bounds for discrete memoryless broadcast channels with confidential messages. Two private messages as well as a common message are transmitted; the common message is to be decoded by both receivers, while each private message is only for its intended receiver. In addition, each private message is to be kept secret from the unintended receiver where secrecy is measured by equivocation. We propose both inner and outer bounds to the rate equivocation region for broadcast channels with confidential messages. The proposed inner bound generalizes Csisz\'{a}r and K\"{o}rner's rate equivocation region for broadcast channels with a single confidential message, Liu {\em et al}'s achievable rate region for broadcast channels with perfect secrecy, Marton's and Gel'fand and Pinsker's achievable rate region for general broadcast channels. Our proposed outer bounds, together with the inner bound, helps establish the rate equivocation region of several classes of discrete memoryless broadcast channels with confidential messages, including less noisy, deterministic, and semi-deterministic channels. Furthermore, specializing to the general broadcast channel by removing the confidentiality constraint, our proposed outer bounds reduce to new capacity outer bounds for the discrete memory broadcast channel.Comment: 27 pages, 1 figure, submitted to IEEE Transaction on Information Theor