Repeatable determinism using non-random weight initialisations in smart city applications of deep learning

Modern Smart City applications draw on the need for requirements that are safe, reliable and sustainable, as such these applications have a need to utilise machine-learning mechanisms such that they are consistent with public liability. Machine and deep learning networks, therefore, are required to be in a form that is safe and deterministic in their development and also in their deployment. The viability of non-random weight initialisation schemes in neural networks make the network more deterministic in learning sessions which is a desirable property in safety critical systems where deep learning is applied to smart city applications and where public liability is a concern. The paper uses a variety of schemes over number ranges and gradients and achieved a 98.09% accuracy figure, + 0.126% higher than the original random number scheme at 97.964%. The paper highlights that in this case, it is the number range and not the gradient that is affecting the achieved accuracy most dominantly, although there can be a coupling of number range with activation functions used. Unexpectedly in this paper, an effect of numerical instability was discovered from run to run when run on a multi-core CPU. The paper also has shown the enforcement of consistent deterministic results on an multi-core CPU by defining atomic critical code regions, and that aids repeatable information assurance in model fitting (or learning sessions). That enforcement of consistent repeatable determinism has also a benefit to accuracy even for the random schemes, and a highest score of 98.29%, + 0.326% higher than the baseline was achieved. However, also the non-random initialisation scheme causes weight arrangements after learning to be more structured which has benefits for validation in safety critical applications

Non-random weight initialisation in deep learning networks for repeatable determinism

This research is examining the change in weight values of deep learning networks after learning. These research experiments require to make measurements and comparisons from a stable set of known weights and biases before and after learning is conducted, such that comparisons after learning are repeatable and the experiment is controlled. As such the current accepted schemes of random number initialisations of the weight values may need to be deterministic rather than stochastic to have little run to run varying effects, so that the weight value initialisations are not a varying contributor. This paper looks at the viability of non-random weight initialisation schemes, to be used in place of the random number weight initialisations of an established well understood test case. The viability of non-random weight initialisation schemes in neural networks may make a network more deterministic in learning sessions which is a desirable property in mission and safety critical systems. The paper will use a variety of schemes over number ranges and gradients and will achieve a 97.97% accuracy figure just 0.18% less than the original random number scheme at 98.05%. The paper may highlight that in this case it may be the number range and not the gradient that is effecting the achieved accuracy most dominantly, although there may be a coupling of number range with activation functions used. Unexpectedly in this paper, an effect of numerical instability will be discovered from run to run when run on a multi-core CPU. The paper will also show the enforcement of consistent deterministic results on an multi-core CPU by defining atomic critical code regions aiding repeatable Information Assurance (IA) in model fitting (or learning sessions)

An algebraic expert system with neural network concepts for cyber, big data and data migration

This paper describes a machine assistance approach to grading decisions for values that might be missing or need validation, using a mathematical algebraic form of an Expert System, instead of the traditional textual or logic forms and builds a neural network computational graph structure. This Experts System approach is also structured into a neural network like format of: input, hidden and output layers that provide a structured approach to the knowledge-base organization, this provides a useful abstraction for reuse for data migration applications in big data, Cyber and relational databases. The approach is further enhanced with a Bayesian probability tree approach to grade the confidences of value probabilities, instead of the traditional grading of the rule probabilities, and estimates the most probable value in light of all evidence presented. This is ground work for a Machine Learning (ML) experts system approach in a form that is closer to a Neural Network node structure

Numerical discrimination of the generalisation model from learnt weights in neural networks

This research demonstrates a method of discriminating the numerical relationships of neural network layer inputs to the layer outputs established from the learnt weights and biases of a neural network's generalisation model. It is demonstrated with a mathematical form of a neural network rather than an image, speech or textual translation application as this provides clarity in the understanding gained from the generalisation model. It is also reliant on the input format but that format is not unlike an image pixel input format and as such the research is applicable to other applications too. The research results have shown that weight and biases can be used to discriminate the mathematical relationships between inputs and make discriminations of what mathematical operators are used between them in the learnt generalisation model. This may be a step towards gaining definitions and understanding for intractable problems that a Neural Network has generalised in a solution. For validating them, or as a mechanism for creating a model used as an alternative to traditional approaches, but derived from a neural network approach as a development tool for solving those problems. The demonstrated method was optimised using learning rate and the number of nodes and in this example achieves a low loss at 7.6e-6, a low Mean Absolute Error at 1e-3 with a high accuracy score of 1.0. But during the experiments a sensitivity to the number of epochs and the use of the random shuffle was discovered, and a comparison with an alternative shuffle using a non-random reordering demonstrated a lower but comparable performance, and is a subject for further research but demonstrated in this "decomposition" class architecture

Deep ConvNet: Non-random weight initialization for repeatable determinism, examined with FSGM

A repeatable and deterministic non-random weight initialization method in convolutional layers of neural networks examined with the Fast Gradient Sign Method (FSGM). Using the FSGM approach as a technique to measure the initialization effect with controlled distortions in transferred learning, varying the dataset numerical similarity. The focus is on convolutional layers with induced earlier learning through the use of striped forms for image classification. Which provided a higher performing accuracy in the first epoch, with improvements of between 3–5% in a well known benchmark model, and also ~10% in a color image dataset (MTARSI2), using a dissimilar model architecture. The proposed method is robust to limit optimization approaches like Glorot/Xavier and He initialization. Arguably the approach is within a new category of weight initialization methods, as a number sequence substitution of random numbers, without a tether to the dataset. When examined under the FGSM approach with transferred learning, the proposed method when used with higher distortions (numerically dissimilar datasets), is less compromised against the original cross-validation dataset, at ~31% accuracy instead of ~9%. This is an indication of higher retention of the original fitting in transferred learning

Novel design for an ultra high precision 3D micro probe for CMM applications

AbstractThe paper reports on the design and testing of several new piezoresistive ultra high precision 3D microprobes for the use in coordinate measuring machines (CMM). The microprobes are all composed of three primary components a piezoresitive sensing element and stylus with probing sphere. The sensing element consists of a bossed KOH-etched silicon membrane with diffused piezoresistors. Several designs were investigated to increase sensor sensitivity while improving the anisotropy of stiffness. The initial design used a Wheatstone bridge piezoresistor configuration with a solid sensing membrane. Additionally, apertures where added to the solid silicon membrane to increase stress within the piezoresistors which untimely led to higher probe tip sensitivity. To improve the stiffness in the xy direction of the probe tip a double triangle design was tested that bonded two sensor chips back to back. This was found reduce the ratio between the stiffness in xy- and z-direction from 32 initially to 2

Travelling with golf clubs: the influence of baggage on the trip decision-making process

Sports participation often requires the use of specialist equipment and for many sport tourists this is transported to the destination to aid convenience and enjoyment of participation. Yet, to date there has been little consideration of the influence that travelling with sporting equipment can have on the trip decisions making process. This paper focuses on golf tourism, said to be the largest sector of the sports tourism market and examines the influence that traveling with golf equipment has on aspects of the trip such as travel mode and opportunities for participation. Based on a longitudinal grounded theory study this paper concludes that packing sporting equipment can stimulate negotiations associated with participation. Furthermore the nature of the sporting equipment to be carried can determine the choices made regarding the travel modes used to reach and move around holiday destinations and thus directly influence the trip decision making process

Aircraft protection for complex threat platforms through integrated EWOS application

The advancement of threats is focusing commanders to consider how to combat these complex threats throughout the kill chain including prior to launch and Anti-Access Area Denial (A2AD). To match this technology race, modern platforms have been designed with integrated command and control systems and automated Defensive Aids Suites built on modular open system architectures incorporating less diverse but more complex software driven systems. The successful operation of these combat systems is reliant upon the availability of accurate, configured, harmonised and “time sensitive” mission data without which the systems may be ineffective. This paper explores the use of threat analysis diagramming techniques, and open architectures tightly integrated with a EWOS life-cycle; to develop countermeasures with a measured response beyond the traditional self-protection kill chain stages of self-protection. It introduces how threat analysis prepares understanding for simulation and how a countermeasure description language can be used to store and exchange countermeasures in a structured form. This level of intelligence data support and analysis coordinated and synchronised across multiple platforms thereby facilitating the complexities of force protection higher up the kill chain into an onion of protection mapped to a Venn diagram of countermeasure types (design intentions) with differing data needs

Non-random weight initialisation in deep convolutional networks applied to safety critical artificial intelligence

This paper presents a non-random weight initialisation scheme for convolutional neural network layers. It builds upon previous work that was limited to perceptron layers, but in that work repeatable determinism was achieved with equality in categorisation accuracy between the established random scheme and a linear ramp non-random scheme. This work however, is in Convolutional layers and are the layers that have been responsible for better than human performance in image recognition. The previous perceptron work found that number range was more important rather than the gradient. However, that was due to the fully connected nature of dense layers. Although, in convolutional layers by contrast, there is an order direction implied, and the weights relate to filters rather than image pixel positions, so the weight initialisation is more complex. However, the paper demonstrates a better performance, over the currently established random schemes with convolutional layers. The proposed method also induces earlier learning through the use of striped forms, and as such has less unlearning of the traditionally speckled random forms. That proposed scheme also provides a higher performing accuracy in a single learning session, with improvements of: 3.35% un-shuffled, 2.813% shuffled in the first epoch and 0.521% over the 5 epochs of the model. Of which the first epoch is more relevant as it is the epoch after initialisation. Also the proposed method is repeatable and deterministic, which is also a desirable quality for safety critical applications within image classification. The proposed method is also robust to He initialisation values too, and scored 97.55% accuracy compared to 96.929% accuracy with the Glorot/ Xavier in the traditional random forms, of which the benchmark model was originally optimised with