Optimizing a Certified Proof Checker for a Large-Scale Computer-Generated Proof

In recent work, we formalized the theory of optimal-size sorting networks with the goal of extracting a verified checker for the large-scale computer-generated proof that 25 comparisons are optimal when sorting 9 inputs, which required more than a decade of CPU time and produced 27 GB of proof witnesses. The checker uses an untrusted oracle based on these witnesses and is able to verify the smaller case of 8 inputs within a couple of days, but it did not scale to the full proof for 9 inputs. In this paper, we describe several non-trivial optimizations of the algorithm in the checker, obtained by appropriately changing the formalization and capitalizing on the symbiosis with an adequate implementation of the oracle. We provide experimental evidence of orders of magnitude improvements to both runtime and memory footprint for 8 inputs, and actually manage to check the full proof for 9 inputs.Comment: IMADA-preprint-c

Structurally dynamic spin market networks

The agent-based model of stock price dynamics on a directed evolving complex network is suggested and studied by direct simulation. The stationary regime is maintained as a result of the balance between the extremal dynamics, adaptivity of strategic variables and reconnection rules. The inherent structure of node agent "brain" is modeled by a recursive neural network with local and global inputs and feedback connections. For specific parametric combination the complex network displays small-world phenomenon combined with scale-free behavior. The identification of a local leader (network hub, agent whose strategies are frequently adapted by its neighbors) is carried out by repeated random walk process through network. The simulations show empirically relevant dynamics of price returns and volatility clustering. The additional emerging aspects of stylized market statistics are Zipfian distributions of fitness.Comment: 13 pages, 5 figures, accepted in IJMPC, references added, minor changes in model, new results and modified figure

A Survey of Adaptive Resonance Theory Neural Network Models for Engineering Applications

This survey samples from the ever-growing family of adaptive resonance theory (ART) neural network models used to perform the three primary machine learning modalities, namely, unsupervised, supervised and reinforcement learning. It comprises a representative list from classic to modern ART models, thereby painting a general picture of the architectures developed by researchers over the past 30 years. The learning dynamics of these ART models are briefly described, and their distinctive characteristics such as code representation, long-term memory and corresponding geometric interpretation are discussed. Useful engineering properties of ART (speed, configurability, explainability, parallelization and hardware implementation) are examined along with current challenges. Finally, a compilation of online software libraries is provided. It is expected that this overview will be helpful to new and seasoned ART researchers

Neural Architecture for Question Answering Using a Knowledge Graph and Web Corpus

In Web search, entity-seeking queries often trigger a special Question Answering (QA) system. It may use a parser to interpret the question to a structured query, execute that on a knowledge graph (KG), and return direct entity responses. QA systems based on precise parsing tend to be brittle: minor syntax variations may dramatically change the response. Moreover, KG coverage is patchy. At the other extreme, a large corpus may provide broader coverage, but in an unstructured, unreliable form. We present AQQUCN, a QA system that gracefully combines KG and corpus evidence. AQQUCN accepts a broad spectrum of query syntax, between well-formed questions to short `telegraphic' keyword sequences. In the face of inherent query ambiguities, AQQUCN aggregates signals from KGs and large corpora to directly rank KG entities, rather than commit to one semantic interpretation of the query. AQQUCN models the ideal interpretation as an unobservable or latent variable. Interpretations and candidate entity responses are scored as pairs, by combining signals from multiple convolutional networks that operate collectively on the query, KG and corpus. On four public query workloads, amounting to over 8,000 queries with diverse query syntax, we see 5--16% absolute improvement in mean average precision (MAP), compared to the entity ranking performance of recent systems. Our system is also competitive at entity set retrieval, almost doubling F1 scores for challenging short queries.Comment: Accepted to Information Retrieval Journa

Modelling the structure of Australian Wool Auction prices

The largest wool exporter in the world is Australia, where wool being a major export is worth over AUD $2 billion per year and constitutes about 17 per cent of all agricultural exports. Most Australian wool is sold by auctions in three regional centres. The prices paid in these auction markets are used by the Australian production and service sectors to identify the quality preferences of the international retail markets and the intermediate processors. One ongoing problem faced by wool growers has been the lack of clear market signals on the relative importance of wool attributes with respect to the price they receive at auction. The goal of our research is to model the structure of Australian wool auction prices. We aim to optimise the information that can be extracted and used by the production and service sectors in producing and distributing the raw wool clip.Most of the previous methods of modelling and predicting wool auction prices employed by the industry have involved multiple-linear regressions. These methods have proven to be inadequate because they have too many assumptions and deficiencies. This has prompted alternative approaches such as neural networks and tree-based regression methods. In this thesis we discuss these alternative approaches. We observe that neural network methods offer good prediction accuracy of price but give minimal understanding of the price driving variables. On the other hand, tree-based regression methods offer good interpretability of the price driving characteristics but do not give good prediction accuracy of price. This motivates a hybrid approach that combines the best of the tree-based methods and neural networks, offering both prediction accuracy and interpretability.Additionally, there also exists a wool specifications problem. Industrial sorting of wool during harvest, and at the start of processing, assembles wool in bins according to the required wool specifications. At present this assembly is done by constraining the range of all specifications in each bin, and having either a very large number of bins, or a large variance of characteristics within each bin. Multiple-linear regression on price does not provide additional useful information that would streamline this process, nor does it assist in delineating the specifications of individual bins.In this thesis we will present a hybrid modular approach combining the interpretability of a regression tree with the prediction accuracy of neural networks. Our procedure was inspired by Breiman and Shang’s idea of a “representer tree” (also known as a “born again tree”) but with two main modifications: 1) we use a much more accurate Neural Network in place of a multiple tree method, and 2) we use our own modified smearing method which involves adding Gaussian noise. Our methodology has not previously been used for wool auction data and the accompanying price prediction problem. The numeric predictions from our method are highly competitive with other methods. Our method also provides an unprecedented level of clarity and interpretability of the price driving variables in the form of tree diagrams, and the tabular form of these trees developed in our research. These are extremely useful for wool growers and other casual observers who may not have a higher level understanding of modelling and mathematics. This method is also highly modular and can be continually extended and improved. We will detail this approach and illustrate it with real data.The more accurate modelling and analysis helps wool growers to better understand the market behaviour. If the important factors are identified, then effective strategies can be developed to maximise return to the growers.In Chapter 1 of this thesis, we present a brief overview of the Australian wool auction market. We then discuss the problems faced by the wool growers and their significance, which motivate our research.In Chapter 2, we define the predictive aspect of the modelling problem and present the data that is available to us for our research. We introduce the assumptions that must be made in order to model the auction data and predict the wool prices.Chapter 3 discusses neural networks and their potential in our wool auction problem. Neural networks are known to give good results in many modern applications resolving industrial problems. As a result of the popularity of such methods and the ongoing development of them, our research partner, the Department of Agriculture and Food, Government of Western Australia, performed a preliminary investigation into neural networks and found them to give satisfactory predictions of wool auction prices. In our Chapter 3, we perform an analysis and assessment of neural networks, specifically, the generalised regression neural networks (GRNN). We look at the strengths and weaknesses of GRNN, and apply them to the wool auction problem and comment on their relevance and usability in our wool problem. We detail the problems we face, and why neural networks alone may not be the best approach for the wool auction problem, thus laying the foundation for the development of our hybrid modular approach in Chapter 5. We also use the numerical prediction results from GRNN as the benchmark in our comparisons of different modelling methods in the rest of this thesis.Chapter 4 details the tree-based regression methods, as an alternate approach to neural networks. In analysing the tree-based methods with our wool auction data, we illustrate the tree methods’ advantages over neural networks, as well as the trade-offs, with our auction data. We also demonstrate how powerful and useful a tree diagram can be to the wool auction problem. And in this Chapter, we improve a typical tree diagram further by introducing our own tabular form of the tree, which can be of immerse use to wool growers. In particular, we can use our tabular form to solve the wool specification problem mentioned earlier, and we incorporate this tabular form as part of a new hybrid methodology in Chapter 5. In Chapter 4 we also consider the ensemble methods such as bootstrap aggregating (bagging) and random forests, and discuss their results. We demonstrate that, the ensemble methods provide higher prediction accuracies than ordinary regression trees by introducing many trees into the model. But this is at the expense of losing the simplicity and clarity of having only a single tree. However, the study of assemble methods do end up providing an excellent idea for our hybrid approach in Chapter 5.Chapter 5 details the new hybrid approach we developed as a result of our work in Chapters 3 and 4 using neural networks and tree-based regression methods. Our hybrid approach combines the two methods with their respective strengths. We apply our new approach to the data, compare the results with our earlier work in neural networks and tree-based regression methods, then discuss the results.Finally, we conclude our thesis with Chapter 6, discussing the potential of our new hybrid approach and the directions of possible future works

Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference

Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. Common neural network compression techniques such as knowledge distillation or quantization are limited to static compression where the compression ratio is fixed. In this paper, we introduce Modular Transformers, a modularized encoder-decoder framework for flexible sequence-to-sequence model compression. Modular Transformers train modularized layers that have the same function of two or more consecutive layers in the original model via module replacing and knowledge distillation. After training, the modularized layers can be flexibly assembled into sequence-to-sequence models that meet different performance-efficiency trade-offs. Experimental results show that after a single training phase, by simply varying the assembling strategy, Modular Transformers can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop.Comment: ACL 2023 Finding

Efficient Error-Tolerant Quantized Neural Network Accelerators

Neural Networks are currently one of the most widely deployed machine learning algorithms. In particular, Convolutional Neural Networks (CNNs), are gaining popularity and are evaluated for deployment in safety critical applications such as self driving vehicles. Modern CNNs feature enormous memory bandwidth and high computational needs, challenging existing hardware platforms to meet throughput, latency and power requirements. Functional safety and error tolerance need to be considered as additional requirement in safety critical systems. In general, fault tolerant operation can be achieved by adding redundancy to the system, which is further exacerbating the computational demands. Furthermore, the question arises whether pruning and quantization methods for performance scaling turn out to be counterproductive with regards to fail safety requirements. In this work we present a methodology to evaluate the impact of permanent faults affecting Quantized Neural Networks (QNNs) and how to effectively decrease their effects in hardware accelerators. We use FPGA-based hardware accelerated error injection, in order to enable the fast evaluation. A detailed analysis is presented showing that QNNs containing convolutional layers are by far not as robust to faults as commonly believed and can lead to accuracy drops of up to 10%. To circumvent that, we propose two different methods to increase their robustness: 1) selective channel replication which adds significantly less redundancy than used by the common triple modular redundancy and 2) a fault-aware scheduling of processing elements for folded implementationsComment: 6 pages, 5 figure