11 research outputs found
Extending the Tsetlin Machine With Integer-Weighted Clauses for Increased Interpretability
Despite significant effort, building models that are both interpretable and
accurate is an unresolved challenge for many pattern recognition problems. In
general, rule-based and linear models lack accuracy, while deep learning
interpretability is based on rough approximations of the underlying inference.
Using a linear combination of conjunctive clauses in propositional logic,
Tsetlin Machines (TMs) have shown competitive performance on diverse
benchmarks. However, to do so, many clauses are needed, which impacts
interpretability. Here, we address the accuracy-interpretability challenge in
machine learning by equipping the TM clauses with integer weights. The
resulting Integer Weighted TM (IWTM) deals with the problem of learning which
clauses are inaccurate and thus must team up to obtain high accuracy as a team
(low weight clauses), and which clauses are sufficiently accurate to operate
more independently (high weight clauses). Since each TM clause is formed
adaptively by a team of Tsetlin Automata, identifying effective weights becomes
a challenging online learning problem. We address this problem by extending
each team of Tsetlin Automata with a stochastic searching on the line (SSL)
automaton. In our novel scheme, the SSL automaton learns the weight of its
clause in interaction with the corresponding Tsetlin Automata team, which, in
turn, adapts the composition of the clause by the adjusting weight. We evaluate
IWTM empirically using five datasets, including a study of interpetability. On
average, IWTM uses 6.5 times fewer literals than the vanilla TM and 120 times
fewer literals than a TM with real-valued weights. Furthermore, in terms of
average F1-Score, IWTM outperforms simple Multi-Layered Artificial Neural
Networks, Decision Trees, Support Vector Machines, K-Nearest Neighbor, Random
Forest, XGBoost, Explainable Boosting Machines, and standard and real-value
weighted TMs.Comment: 20 pages, 10 figure
Active classification with comparison queries
We study an extension of active learning in which the learning algorithm may
ask the annotator to compare the distances of two examples from the boundary of
their label-class. For example, in a recommendation system application (say for
restaurants), the annotator may be asked whether she liked or disliked a
specific restaurant (a label query); or which one of two restaurants did she
like more (a comparison query).
We focus on the class of half spaces, and show that under natural
assumptions, such as large margin or bounded bit-description of the input
examples, it is possible to reveal all the labels of a sample of size using
approximately queries. This implies an exponential improvement over
classical active learning, where only label queries are allowed. We complement
these results by showing that if any of these assumptions is removed then, in
the worst case, queries are required.
Our results follow from a new general framework of active learning with
additional queries. We identify a combinatorial dimension, called the
\emph{inference dimension}, that captures the query complexity when each
additional query is determined by examples (such as comparison queries,
each of which is determined by the two compared examples). Our results for half
spaces follow by bounding the inference dimension in the cases discussed above.Comment: 23 pages (not including references), 1 figure. The new version
contains a minor fix in the proof of Lemma 4.
Minimizing DNF Formulas and AC 0 Circuits Given a Truth Table
For circuit classes R, the fundamental computational problem Min-R asks for the minimum R-size of a Boolean function presented as a truth table. Prominent examples of this problem include Min-DNF, which asks whether a given Boolean function presented as a truth table has a k-term DNF, and Min-Circuit (also called MCSP), which asks whether a Boolean function presented as a truth table has a size k Boolean circuit. We present a new reduction proving that Min-DNF is NP-complete. It is significantly simpler than the known reduction of Masek [31], which is from Circuit-SAT. We then give a more complex reduction, yielding the result that Min-DNF cannot be approximated to within a factor smaller than logN γ, for some constant γ 0, assuming that NP is not contained in quasipolynomial time. The standard greedy algorithm for Set Cover is often used in practice to approximate Min-DNF. The question of whether Min-DNF can be approximated to within a factor of o logN remains open, but we construct an instance of Min-DNF on which the solution produced by the greedy algorithm is Ω logN larger than optimal. Finally, we extend known hardness results for Min-TC0 d to obtain new hardness results for Min-AC0 d, under cryptographic assumptions
Hardness of Approximate Two-level Logic Minimization and PAC Learning with Membership Queries
Producing a small DNF expression consistent with given data is a classical problem in computer science that occurs in a number of forms and has numerous applications. We consider two standard variants of this problem. The first one is two-level logic minimization or finding a minimum DNF formula consistent with a given complete truth table (TT-MinDNF). This problem was formulated by Quine in 1952 and has been since one of the key problems in logic design. It was proved NP-complete by Masek in 1979. The best known polynomial approximation algorithm is based on a reduction to the SET-COVER problem and produces a DNF formula of size O(d · OPT), where d is the number of variables. We prove that TT-MinDNF is NP-hard to approximate within d γ for some constant γ> 0, establishing the first inapproximability result for the problem. The other DNF minimization problem we consider is PAC learning of DNF expressions when the learning algorithm must output a DNF expression as its hypothesis (referred to as proper learning). We prove that DNF expressions are NP-hard to PAC learn properly even when the learner has access to membership queries, thereby answering a long-standing open question due to Valiant [40]. Finally, we provide a concrete connection between these variants of DNF minimization problem. Specifically, we prove that inapproximability of TT-MinDNF implies hardness results for restricted proper learning of DNF expressions with membership queries even when learning with respect to the uniform distribution only
Hardness of approximate two-level logic minimization and pac learning with membership queries
Producing a small DNF expression consistent with given data is a classical problem in computer science that occurs in a number of forms and has numerous applications. We consider two standard variants of this problem. The first one is two-level logic minimization or finding a minimum DNF formula consistent with a given complete truth table (TT-MinDNF). This problem was formulated by Quine in 1952 and has been since one of the key problems in logic design. It was proved NP-complete by Masek in 1979. The best known polynomial approximation algorithm is based on a reduction to the SET-COVER problem and produces a DNF formula of size O(d · OPT), where d is the number of variables. We prove that TT-MinDNF is NP-hard to approximate within d γ for some constant γ> 0, establishing the first inapproximability result for the problem. The other DNF minimization problem we consider is PAC learning of DNF expressions when the learning algorithm must output a DNF expression as its hypothesis (referred to as proper learning). We prove that DNF expressions are NP-hard to PAC learn properly even when the learner has access to membership queries, thereby answering a long-standing open question due to Valiant [40]. Finally, we provide a concrete connection between these variants of DNF minimization problem. Specifically, we prove that inapproximability of TT-MinDNF implies hardness results for restricted proper learning of DNF expressions with membership queries even when learning with respect to the uniform distribution only