Analysis of classifiers' robustness to adversarial perturbations

The goal of this paper is to analyze an intriguing phenomenon recently discovered in deep networks, namely their instability to adversarial perturbations (Szegedy et. al., 2014). We provide a theoretical framework for analyzing the robustness of classifiers to adversarial perturbations, and show fundamental upper bounds on the robustness of classifiers. Specifically, we establish a general upper bound on the robustness of classifiers to adversarial perturbations, and then illustrate the obtained upper bound on the families of linear and quadratic classifiers. In both cases, our upper bound depends on a distinguishability measure that captures the notion of difficulty of the classification task. Our results for both classes imply that in tasks involving small distinguishability, no classifier in the considered set will be robust to adversarial perturbations, even if a good accuracy is achieved. Our theoretical framework moreover suggests that the phenomenon of adversarial instability is due to the low flexibility of classifiers, compared to the difficulty of the classification task (captured by the distinguishability). Moreover, we show the existence of a clear distinction between the robustness of a classifier to random noise and its robustness to adversarial perturbations. Specifically, the former is shown to be larger than the latter by a factor that is proportional to \sqrt{d} (with d being the signal dimension) for linear classifiers. This result gives a theoretical explanation for the discrepancy between the two robustness properties in high dimensional problems, which was empirically observed in the context of neural networks. To the best of our knowledge, our results provide the first theoretical work that addresses the phenomenon of adversarial instability recently observed for deep networks. Our analysis is complemented by experimental results on controlled and real-world data

Algorithmic Aspects of Optimal Channel Coding

A central question in information theory is to determine the maximum success probability that can be achieved in sending a fixed number of messages over a noisy channel. This was first studied in the pioneering work of Shannon who established a simple expression characterizing this quantity in the limit of multiple independent uses of the channel. Here we consider the general setting with only one use of the channel. We observe that the maximum success probability can be expressed as the maximum value of a submodular function. Using this connection, we establish the following results: 1. There is a simple greedy polynomial-time algorithm that computes a code achieving a (1-1/e)-approximation of the maximum success probability. Moreover, for this problem it is NP-hard to obtain an approximation ratio strictly better than (1-1/e). 2. Shared quantum entanglement between the sender and the receiver can increase the success probability by a factor of at most 1/(1-1/e). In addition, this factor is tight if one allows an arbitrary non-signaling box between the sender and the receiver. 3. We give tight bounds on the one-shot performance of the meta-converse of Polyanskiy-Poor-Verdu.Comment: v2: 16 pages. Added alternate proof of main result with random codin

Scrambling speed of random quantum circuits

Random transformations are typically good at "scrambling" information. Specifically, in the quantum setting, scrambling usually refers to the process of mapping most initial pure product states under a unitary transformation to states which are macroscopically entangled, in the sense of being close to completely mixed on most subsystems containing a fraction fn of all n particles for some constant f. While the term scrambling is used in the context of the black hole information paradox, scrambling is related to problems involving decoupling in general, and to the question of how large isolated many-body systems reach local thermal equilibrium under their own unitary dynamics. Here, we study the speed at which various notions of scrambling/decoupling occur in a simplified but natural model of random two-particle interactions: random quantum circuits. For a circuit representing the dynamics generated by a local Hamiltonian, the depth of the circuit corresponds to time. Thus, we consider the depth of these circuits and we are typically interested in what can be done in a depth that is sublinear or even logarithmic in the size of the system. We resolve an outstanding conjecture raised in the context of the black hole information paradox with respect to the depth at which a typical quantum circuit generates an entanglement assisted encoding against the erasure channel. In addition, we prove that typical quantum circuits of poly(log n) depth satisfy a stronger notion of scrambling and can be used to encode alpha n qubits into n qubits so that up to beta n errors can be corrected, for some constants alpha, beta > 0.Comment: 24 pages, 2 figures. Superseded by http://arxiv.org/abs/1307.063

On simultaneous min-entropy smoothing

In the context of network information theory, one often needs a multiparty probability distribution to be typical in several ways simultaneously. When considering quantum states instead of classical ones, it is in general difficult to prove the existence of a state that is jointly typical. Such a difficulty was recently emphasized and conjectures on the existence of such states were formulated. In this paper, we consider a one-shot multiparty typicality conjecture. The question can then be stated easily: is it possible to smooth the largest eigenvalues of all the marginals of a multipartite state {\rho} simultaneously while staying close to {\rho}? We prove the answer is yes whenever the marginals of the state commute. In the general quantum case, we prove that simultaneous smoothing is possible if the number of parties is two or more generally if the marginals to optimize satisfy some non-overlap property.Comment: 5 page

Decoupling with random quantum circuits

Decoupling has become a central concept in quantum information theory with applications including proving coding theorems, randomness extraction and the study of conditions for reaching thermal equilibrium. However, our understanding of the dynamics that lead to decoupling is limited. In fact, the only families of transformations that are known to lead to decoupling are (approximate) unitary two-designs, i.e., measures over the unitary group which behave like the Haar measure as far as the first two moments are concerned. Such families include for example random quantum circuits with O(n^2) gates, where n is the number of qubits in the system under consideration. In fact, all known constructions of decoupling circuits use \Omega(n^2) gates. Here, we prove that random quantum circuits with O(n log^2 n) gates satisfy an essentially optimal decoupling theorem. In addition, these circuits can be implemented in depth O(log^3 n). This proves that decoupling can happen in a time that scales polylogarithmically in the number of particles in the system, provided all the particles are allowed to interact. Our proof does not proceed by showing that such circuits are approximate two-designs in the usual sense, but rather we directly analyze the decoupling property.Comment: 25 page

Short random circuits define good quantum error correcting codes

We study the encoding complexity for quantum error correcting codes with large rate and distance. We prove that random Clifford circuits with $O(n \log^2 n)$ gates can be used to encode $k$ qubits in $n$ qubits with a distance $d$ provided $\frac{k}{n} < 1 - \frac{d}{n} \log_2 3 - h(\frac{d}{n})$. In addition, we prove that such circuits typically have a depth of $O( \log^3 n)$.Comment: 5 page

Longest path distance in random circuits

We study distance properties of a general class of random directed acyclic graphs (DAGs). In a DAG, many natural notions of distance are possible, for there exists multiple paths between pairs of nodes. The distance of interest for circuits is the maximum length of a path between two nodes. We give laws of large numbers for the typical depth (distance to the root) and the minimum depth in a random DAG. This completes the study of natural distances in random DAGs initiated (in the uniform case) by Devroye and Janson (2009+). We also obtain large deviation bounds for the minimum of a branching random walk with constant branching, which can be seen as a simplified version of our main result.Comment: 21 pages, 2 figure

Entanglement sampling and applications

A natural measure for the amount of quantum information that a physical system E holds about another system A = A_1,...,A_n is given by the min-entropy Hmin(A|E). Specifically, the min-entropy measures the amount of entanglement between E and A, and is the relevant measure when analyzing a wide variety of problems ranging from randomness extraction in quantum cryptography, decoupling used in channel coding, to physical processes such as thermalization or the thermodynamic work cost (or gain) of erasing a quantum system. As such, it is a central question to determine the behaviour of the min-entropy after some process M is applied to the system A. Here we introduce a new generic tool relating the resulting min-entropy to the original one, and apply it to several settings of interest, including sampling of subsystems and measuring in a randomly chosen basis. The sampling results lead to new upper bounds on quantum random access codes, and imply the existence of "local decouplers". The results on random measurements yield new high-order entropic uncertainty relations with which we prove the optimality of cryptographic schemes in the bounded quantum storage model.Comment: v3: fixed some typos, v2: fixed minor issue with the definition of entropy and improved presentatio

On Variational Expressions for Quantum Relative Entropies

Distance measures between quantum states like the trace distance and the fidelity can naturally be defined by optimizing a classical distance measure over all measurement statistics that can be obtained from the respective quantum states. In contrast, Petz showed that the measured relative entropy, defined as a maximization of the Kullback-Leibler divergence over projective measurement statistics, is strictly smaller than Umegaki's quantum relative entropy whenever the states do not commute. We extend this result in two ways. First, we show that Petz' conclusion remains true if we allow general positive operator valued measures. Second, we extend the result to Renyi relative entropies and show that for non-commuting states the sandwiched Renyi relative entropy is strictly larger than the measured Renyi relative entropy for $\alpha \in (\frac12, \infty)$, and strictly smaller for $\alpha \in [0,\frac12)$. The latter statement provides counterexamples for the data-processing inequality of the sandwiched Renyi relative entropy for $\alpha < \frac12$. Our main tool is a new variational expression for the measured Renyi relative entropy, which we further exploit to show that certain lower bounds on quantum conditional mutual information are superadditive.Comment: v2: final published versio