Relative Entropy and Inductive Inference

We discuss how the method of maximum entropy, MaxEnt, can be extended beyond its original scope, as a rule to assign a probability distribution, to a full-fledged method for inductive inference. The main concept is the (relative) entropy S[p|q] which is designed as a tool to update from a prior probability distribution q to a posterior probability distribution p when new information in the form of a constraint becomes available. The extended method goes beyond the mere selection of a single posterior p, but also addresses the question of how much less probable other distributions might be. Our approach clarifies how the entropy S[p|q] is used while avoiding the question of its meaning. Ultimately, entropy is a tool for induction which needs no interpretation. Finally, being a tool for generalization from special examples, we ask whether the functional form of the entropy depends on the choice of the examples and we find that it does. The conclusion is that there is no single general theory of inductive inference and that alternative expressions for the entropy are possible.Comment: Presented at MaxEnt23, the 23rd International Workshop on Bayesian Inference and Maximum Entropy Methods (August 3-8, 2003, Jackson Hole, WY, USA

Subjectivity in inductive inference

This paper examines circumstances under which subjectivity enhances the effectiveness of inductive reasoning. We consider agents facing a data generating process who are characterized by inference rules that may be purely objective (or data-based) or may incorporate subjective considerations. The basic intuition is that agents who invoke no subjective considerations are doomed to "overfit" the data and therefore engage in ineffective learning. The analysis places no computational or memory limitations on the agents|the role for subjectivity emerges in the presence of unlimited reasoning powers.Inductive inference, simplicity, prediction, learning

Inductive Inference: An Axiomatic Approach

A predictor is asked to rank eventualities according to their plausibility, based on past cases. We assume that she can form a ranking given any memory that consists of finitely many past cases. Mild consistency requirements on these rankings imply that they have a numerical representation via a matrix assigning numbers to eventuality-case pairs, as follows. Given a memory, each eventuality is ranked according to the sum of the numbers in its row, over cases in memory. The number attached to an eventuality-case pair can be interpreted as the degree of support that the past case lends to the plausibility of the eventuality. Special instances of this result may be viewed as axiomatizing kernel methods for estimation of densities and for classification problems. Interpreting the same result for rankings of theories or hypotheses, rather than of specific eventualities, it is shown that one may ascribe to the predictor subjective conditional probabilities of cases given theories, such that her rankings of theories agree with rankings by the likelihood functions.Inductive inference, case-based reasoning,case-based decision theory, maximum likelihood

Sciduction: Combining Induction, Deduction, and Structure for Verification and Synthesis

Even with impressive advances in automated formal methods, certain problems in system verification and synthesis remain challenging. Examples include the verification of quantitative properties of software involving constraints on timing and energy consumption, and the automatic synthesis of systems from specifications. The major challenges include environment modeling, incompleteness in specifications, and the complexity of underlying decision problems. This position paper proposes sciduction, an approach to tackle these challenges by integrating inductive inference, deductive reasoning, and structure hypotheses. Deductive reasoning, which leads from general rules or concepts to conclusions about specific problem instances, includes techniques such as logical inference and constraint solving. Inductive inference, which generalizes from specific instances to yield a concept, includes algorithmic learning from examples. Structure hypotheses are used to define the class of artifacts, such as invariants or program fragments, generated during verification or synthesis. Sciduction constrains inductive and deductive reasoning using structure hypotheses, and actively combines inductive and deductive reasoning: for instance, deductive techniques generate examples for learning, and inductive reasoning is used to guide the deductive engines. We illustrate this approach with three applications: (i) timing analysis of software; (ii) synthesis of loop-free programs, and (iii) controller synthesis for hybrid systems. Some future applications are also discussed

Induction and Natural Kinds Revisited

In ‘Induction and Natural Kinds’, I proposed a solution to the problem of induction according to which our use of inductive inference is reliable because it is grounded in the natural kind structure of the world. When we infer that unobserved members of a kind will have the same properties as observed members of the kind, we are right because all members of the kind possess the same essential properties. The claim that the existence of natural kinds is what grounds reliable use of induction is based on an inference to the best explanation of the success of our inductive practices. As such, the argument for the existence of natural kinds employs a form of ampliative inference. But induction is likewise a form of ampliative inference. Given both of these facts, my account of the reliability of induction is subject to the objection that it provides a circular justification of induction, since it employs an ampliative inference to justify an ampliative inference. In this paper, I respond to the objection of circularity by arguing that what justifies induction is not the inference to the best explanation of its reliability. The ground of induction is the natural kinds themselves

The Problem of Analogical Inference in Inductive Logic

We consider one problem that was largely left open by Rudolf Carnap in his work on inductive logic, the problem of analogical inference. After discussing some previous attempts to solve this problem, we propose a new solution that is based on the ideas of Bruno de Finetti on probabilistic symmetries. We explain how our new inductive logic can be developed within the Carnapian paradigm of inductive logic-deriving an inductive rule from a set of simple postulates about the observational process-and discuss some of its properties.Comment: In Proceedings TARK 2015, arXiv:1606.0729