42,058 research outputs found
Towards Efficient and Trustworthy AI Through Hardware-Algorithm-Communication Co-Design
Artificial intelligence (AI) algorithms based on neural networks have been
designed for decades with the goal of maximising some measure of accuracy. This
has led to two undesired effects. First, model complexity has risen
exponentially when measured in terms of computation and memory requirements.
Second, state-of-the-art AI models are largely incapable of providing
trustworthy measures of their uncertainty, possibly `hallucinating' their
answers and discouraging their adoption for decision-making in sensitive
applications.
With the goal of realising efficient and trustworthy AI, in this paper we
highlight research directions at the intersection of hardware and software
design that integrate physical insights into computational substrates,
neuroscientific principles concerning efficient information processing,
information-theoretic results on optimal uncertainty quantification, and
communication-theoretic guidelines for distributed processing. Overall, the
paper advocates for novel design methodologies that target not only accuracy
but also uncertainty quantification, while leveraging emerging computing
hardware architectures that move beyond the traditional von Neumann digital
computing paradigm to embrace in-memory, neuromorphic, and quantum computing
technologies. An important overarching principle of the proposed approach is to
view the stochasticity inherent in the computational substrate and in the
communication channels between processors as a resource to be leveraged for the
purpose of representing and processing classical and quantum uncertainty
Bits from Biology for Computational Intelligence
Computational intelligence is broadly defined as biologically-inspired
computing. Usually, inspiration is drawn from neural systems. This article
shows how to analyze neural systems using information theory to obtain
constraints that help identify the algorithms run by such systems and the
information they represent. Algorithms and representations identified
information-theoretically may then guide the design of biologically inspired
computing systems (BICS). The material covered includes the necessary
introduction to information theory and the estimation of information theoretic
quantities from neural data. We then show how to analyze the information
encoded in a system about its environment, and also discuss recent
methodological developments on the question of how much information each agent
carries about the environment either uniquely, or redundantly or
synergistically together with others. Last, we introduce the framework of local
information dynamics, where information processing is decomposed into component
processes of information storage, transfer, and modification -- locally in
space and time. We close by discussing example applications of these measures
to neural data and other complex systems
Context-Aware Generative Adversarial Privacy
Preserving the utility of published datasets while simultaneously providing
provable privacy guarantees is a well-known challenge. On the one hand,
context-free privacy solutions, such as differential privacy, provide strong
privacy guarantees, but often lead to a significant reduction in utility. On
the other hand, context-aware privacy solutions, such as information theoretic
privacy, achieve an improved privacy-utility tradeoff, but assume that the data
holder has access to dataset statistics. We circumvent these limitations by
introducing a novel context-aware privacy framework called generative
adversarial privacy (GAP). GAP leverages recent advancements in generative
adversarial networks (GANs) to allow the data holder to learn privatization
schemes from the dataset itself. Under GAP, learning the privacy mechanism is
formulated as a constrained minimax game between two players: a privatizer that
sanitizes the dataset in a way that limits the risk of inference attacks on the
individuals' private variables, and an adversary that tries to infer the
private variables from the sanitized dataset. To evaluate GAP's performance, we
investigate two simple (yet canonical) statistical dataset models: (a) the
binary data model, and (b) the binary Gaussian mixture model. For both models,
we derive game-theoretically optimal minimax privacy mechanisms, and show that
the privacy mechanisms learned from data (in a generative adversarial fashion)
match the theoretically optimal ones. This demonstrates that our framework can
be easily applied in practice, even in the absence of dataset statistics.Comment: Improved version of a paper accepted by Entropy Journal, Special
Issue on Information Theory in Machine Learning and Data Scienc
Function-Theoretic Explanation and the Search for Neural Mechanisms
A common kind of explanation in cognitive neuroscience might be called functiontheoretic:
with some target cognitive capacity in view, the theorist hypothesizes that
the system computes a well-defined function (in the mathematical sense) and explains
how computing this function constitutes (in the system’s normal environment) the
exercise of the cognitive capacity. Recently, proponents of the so-called ‘new mechanist’
approach in philosophy of science have argued that a model of a cognitive capacity is
explanatory only to the extent that it reveals the causal structure of the mechanism
underlying the capacity. If they are right, then a cognitive model that resists a transparent
mapping to known neural mechanisms fails to be explanatory. I argue that a functiontheoretic
characterization of a cognitive capacity can be genuinely explanatory even
absent an account of how the capacity is realized in neural hardware
Information theoretic novelty detection
We present a novel approach to online change detection problems when the training sample size is small. The proposed approach is based on estimating the expected information content of a new data point and allows an accurate control of the false positive rate even for small data sets. In the case of the Gaussian distribution, our approach is analytically tractable and closely related
to classical statistical tests. We then propose an approximation scheme to extend our approach to the case of the mixture of Gaussians. We evaluate extensively our approach on synthetic data and on three real benchmark data
sets. The experimental validation shows that our method maintains a good overall accuracy, but significantly improves the control over the false positive rate
Applications of Information Theory to Analysis of Neural Data
Information theory is a practical and theoretical framework developed for the
study of communication over noisy channels. Its probabilistic basis and
capacity to relate statistical structure to function make it ideally suited for
studying information flow in the nervous system. It has a number of useful
properties: it is a general measure sensitive to any relationship, not only
linear effects; it has meaningful units which in many cases allow direct
comparison between different experiments; and it can be used to study how much
information can be gained by observing neural responses in single trials,
rather than in averages over multiple trials. A variety of information
theoretic quantities are commonly used in neuroscience - (see entry
"Definitions of Information-Theoretic Quantities"). In this entry we review
some applications of information theory in neuroscience to study encoding of
information in both single neurons and neuronal populations.Comment: 8 pages, 2 figure
- …