Ordinal regression methods: survey and experimental study

Abstract—Ordinal regression problems are those machine learning problems where the objective is to classify patterns using a categorical scale which shows a natural order between the labels. Many real-world applications present this labelling structure and that has increased the number of methods and algorithms developed over the last years in this field. Although ordinal regression can be faced using standard nominal classification techniques, there are several algorithms which can specifically benefit from the ordering information. Therefore, this paper is aimed at reviewing the state of the art on these techniques and proposing a taxonomy based on how the models are constructed to take the order into account. Furthermore, a thorough experimental study is proposed to check if the use of the order information improves the performance of the models obtained, considering some of the approaches within the taxonomy. The results confirm that ordering information benefits ordinal models improving their accuracy and the closeness of the predictions to actual targets in the ordinal scal

Essays on distance metric learning

Many machine learning methods, such as the k-nearest neighbours algorithm, heavily depend on the distance measure between data points. As each task has its own notion of distance, distance metric learning has been proposed. It learns a distance metric to assign a small distance to semantically similar instances and a large distance to dissimilar instances by formulating an optimisation problem. While many loss functions and regularisation terms have been proposed to improve the discrimination and generalisation ability of the learned metric, the metric may be sensitive to a small perturbation in the input space. Moreover, these methods implicitly assume that features are numerical variables and labels are deterministic. However, categorical variables and probabilistic labels are common in real-world applications. This thesis develops three metric learning methods to enhance robustness against input perturbation and applicability for categorical variables and probabilistic labels. In Chapter 3, I identify that many existing methods maximise a margin in the feature space and such margin is insufficient to withstand perturbation in the input space. To address this issue, a new loss function is designed to penalise the input-space margin for being small and hence improve the robustness of the learned metric. In Chapter 4, I propose a metric learning method for categorical data. Classifying categorical data is difficult due to high feature ambiguity, and to this end, the technique of adversarial training is employed. Moreover, the generalisation bound of the proposed method is established, which informs the choice of the regularisation term. In Chapter 5, I adapt a classical probabilistic approach for metric learning to utilise information on probabilistic labels. The loss function is modified for training stability, and new evaluation criteria are suggested to assess the effectiveness of different methods. At the end of this thesis, two publications on hyperspectral target detection are appended as additional work during my PhD

Reduction of emergency department returns after discharge from hospital: Machine learning model to predict emergency department returns 30 days post hospital discharge for medical patients

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Business AnalyticsPost-hospital discharge returns to emergency departments are associated with reducing the efficiency of the emergency department (ED) utilisation and the quality of healthcare. These returns are often related to the nature of the disease and/or inadequate care. This thesis aims to develop a machine-learning model that predicts ED returns within 30 days of inpatient discharge from Portuguese public hospitals. Different binary classification models were trained and evaluated with a particular focus on sensitivity (predictive power of the critical class of returning patients). The selected model was the Extreme gradient boost Classifier, which showed the best performance on recall and the other considered performance metrics. A cohort of 93 449 medical hospitalisations of adult patients discharged between January 1st, 2018, and December 31st, 2019, was assembled with diagnoses details to be used in this study. According to the problem's requirement, the recall was the performance metric to be maximised. Therefore, Performance optimisation methods were considered, and the final model resulted in a recall of 84.38%, precision of 84.35%, F1 score of 84.36% and accuracy of 84.10%. Future deployment and integration of this ED return predictive analytics into the inpatient care workflow may allow identifying patients that require targeted care interventions that reduce overall healthcare expense and improve health outcomes

Interactive Evolutionary Algorithms for Image Enhancement and Creation

Image enhancement and creation, particularly for aesthetic purposes, are tasks for which the use of interactive evolutionary algorithms would seem to be well suited. Previous work has concentrated on the development of various aspects of the interactive evolutionary algorithms and their application to various image enhancement and creation problems. Robust evaluation of algorithmic design options in interactive evolutionary algorithms and the comparison of interactive evolutionary algorithms to alternative approaches to achieving the same goals is generally less well addressed. The work presented in this thesis is primarily concerned with different interactive evolutionary algorithms, search spaces, and operators for setting the input values required by image processing and image creation tasks. A secondary concern is determining when the use of the interactive evolutionary algorithm approach to image enhancement problems is warranted and how it compares with alternative approaches. Various interactive evolutionary algorithms were implemented and compared in a number of specifically devised experiments using tasks of varying complexity. A novel aspect of this thesis, with regards to other work in the study of interactive evolutionary algorithms, was that statistical analysis of the data gathered from the experiments was performed. This analysis demonstrated, contrary to popular assumption, that the choice of algorithm parameters, operators, search spaces, and even the underlying evolutionary algorithm has little effect on the quality of the resulting images or the time it takes to develop them. It was found that the interaction methods chosen when implementing the user interface of the interactive evolutionary algorithms had a greater influence on the performances of the algorithms

Assessing the reliability of an automated method for measuring dominance hierarchy in non-human primates

Among animal societies, dominance is an important social factor that influences inter-individual relationships. However, assessing dominance hierarchy can be a time-consuming activity which is potentially impeded by environmental factors, difficulties in the recognition of animals, or disturbance of animals during data collection. Here we took advantage of novel devices, machines for automated learning and testing (MALT), designed primarily to study non-human primate cognition, to additionally measure the dominance hierarchy of a semi-free-ranging primate group. When working on a MALT, an animal can be replaced by another, which could reflect an asymmetric dominance relationship. To assess the reliability of our method, we analysed a sample of the automated conflicts with video scoring and found that 74% of these replacements included genuine forms of social displacements. In 10% of the cases, we did not identify social interactions and in the remaining 16% we observed affiliative contacts between the monkeys. We analysed months of daily use of MALT by up to 26 semi-free-ranging Tonkean macaques (Macaca tonkeana) and found that dominance relationships inferred from these interactions strongly correlated with the ones derived from observations of spontaneous agonistic interactions collected during the same time period. An optional filtering procedure designed to exclude chance-driven displacements or affiliative contacts suggests that the presence of 26% of these interactions in data sets did not impair the reliability of this new method. We demonstrate that this method can be used to assess the dynamics of both individual social status, and group-wide hierarchical stability longitudinally with minimal research labour. Further, it facilitates a continuous assessment of dominance hierarchies in captive groups, even during unpredictable environmental or challenging social events, which underlines the usefulness of this method for group management purposes. Altogether, this study supports the use of MALT as a reliable tool to automatically and dynamically assess dominance hierarchy within captive groups of non-human primates, including juveniles, under conditions in which such technology can be used

The entrepreneurial process of the entrepreneurs in Germany and the role of experiential learning in this process

Diese Dissertation schlägt vor, Unternehmertum durch die Theorie des Erfahrungslernens nach Kolb (2015) zu fördern. Unternehmertum wird erreicht, wenn es dem Unternehmer gelingt, eine Gelegenheit zu erkennen. Unternehmerische Selbstwirksamkeit, innovatives Verhalten und die Wahrnehmung von Chancen sind die Säulen des konzeptionellen Modells dieser Forschung. Die Schätzungen dieses Modells stützten sich auf Primärdaten, die von Unternehmern in Deutschland erhoben wurden (N=309). Für die Datenanalyse und den Test des hypothetischen Modells wird in dieser Studie das Strukturgleichungsmodell (SEM) verwendet. Die Ergebnisse zeigen, dass die unternehmerische Selbstwirksamkeit einen direkten, positiven Effekt auf das Erkennen von Chancen bei deutschen Unternehmern hat. Die Ergebnisse deuten darauf hin, dass innovatives Verhalten die direkte Beziehung zwischen unternehmerischer Selbstwirksamkeit und Chancenwahrnehmung vollständige Mediation vermittelt. Der Einsatz von innovativem Verhalten zur hancenerkennung führt zu einer innovativen Gelegenheit und damit zu einem Mehrwert für die Wirtschaft. Um diesen unternehmerischen Prozess zu unterstützen, schlägt diese Studie für jedes Konstrukt des konzeptionellen Modells einen erfahrungsbasierten Lernstil vor. Die aktuelle Studie erklärt empirisch, warum manche Menschen in Deutschland eine Chance erkennen können, andere aber nicht. Auf diese Weise erklärt sie, warum Unternehmer in Deutschland eine höhere Wertschöpfung erbringen als in den Nachbarländern der Europäischen Union. Des Weiteren leistet sie einen Beitrag zur Unternehmer-Ausbildung, indem sie die Theorie des Erfahrungslernens in einen unternehmerischen Prozess einführt, durch den ein Unternehmer lernt, eine Gelegenheit zu erkennen. In diesem Zusammenhang wird versucht zu beantworten, wie Unternehmer lernen können, eine Chance wahrzunehmen

Agrupamiento, predicción y clasificación ordinal para series temporales utilizando técnicas de machine learning: aplicaciones

In the last years, there has been an increase in the number of fields improving their standard processes by using machine learning (ML) techniques. The main reason for this is that the vast amount of data generated by these processes is difficult to be processed by humans. Therefore, the development of automatic methods to process and extract relevant information from these data processes is of great necessity, giving that these approaches could lead to an increase in the economic benefit of enterprises or to a reduction in the workload of some current employments. Concretely, in this Thesis, ML approaches are applied to problems concerning time series data. Time series is a special kind of data in which data points are collected chronologically. Time series are present in a wide variety of fields, such as atmospheric events or engineering applications. Besides, according to the main objective to be satisfied, there are different tasks in the literature applied to time series. Some of them are those on which this Thesis is mainly focused: clustering, classification, prediction and, in general, analysis. Generally, the amount of data to be processed is huge, arising the need of methods able to reduce the dimensionality of time series without decreasing the amount of information. In this sense, the application of time series segmentation procedures dividing the time series into different subsequences is a good option, given that each segment defines a specific behaviour. Once the different segments are obtained, the use of statistical features to characterise them is an excellent way to maximise the information of the time series and simultaneously reducing considerably their dimensionality. In the case of time series clustering, the objective is to find groups of similar time series with the idea of discovering interesting patterns in time series datasets. In this Thesis, we have developed a novel time series clustering technique. The aim of this proposal is twofold: to reduce as much as possible the dimensionality and to develop a time series clustering approach able to outperform current state-of-the-art techniques. In this sense, for the first objective, the time series are segmented in order to divide the them identifying different behaviours. Then, these segments are projected into a vector of statistical features aiming to reduce the dimensionality of the time series. Once this preprocessing step is done, the clustering of the time series is carried out, with a significantly lower computational load. This novel approach has been tested on all the time series datasets available in the University of East Anglia and University of California Riverside (UEA/UCR) time series classification (TSC) repository. Regarding time series classification, two main paths could be differentiated: firstly, nominal TSC, which is a well-known field involving a wide variety of proposals and transformations applied to time series. Concretely, one of the most popular transformation is the shapelet transform (ST), which has been widely used in this field. The original method extracts shapelets from the original time series and uses them for classification purposes. Nevertheless, the full enumeration of all possible shapelets is very time consuming. Therefore, in this Thesis, we have developed a hybrid method that starts with the best shapelets extracted by using the original approach with a time constraint and then tunes these shapelets by using a convolutional neural network (CNN) model. Secondly, time series ordinal classification (TSOC) is an unexplored field beginning with this Thesis. In this way, we have adapted the original ST to the ordinal classification (OC) paradigm by proposing several shapelet quality measures taking advantage of the ordinal information of the time series. This methodology leads to better results than the state-of-the-art TSC techniques for those ordinal time series datasets. All these proposals have been tested on all the time series datasets available in the UEA/UCR TSC repository. With respect to time series prediction, it is based on estimating the next value or values of the time series by considering the previous ones. In this Thesis, several different approaches have been considered depending on the problem to be solved. Firstly, the prediction of low-visibility events produced by fog conditions is carried out by means of hybrid autoregressive models (ARs) combining fixed-size and dynamic windows, adapting itself to the dynamics of the time series. Secondly, the prediction of convective cloud formation (which is a highly imbalance problem given that the number of convective cloud events is much lower than that of non-convective situations) is performed in two completely different ways: 1) tackling the problem as a multi-objective classification task by the use of multi-objective evolutionary artificial neural networks (MOEANNs), in which the two conflictive objectives are accuracy of the minority class and the global accuracy, and 2) tackling the problem from the OC point of view, in which, in order to reduce the imbalance degree, an oversampling approach is proposed along with the use of OC techniques. Thirdly, the prediction of solar radiation is carried out by means of evolutionary artificial neural networks (EANNs) with different combinations of basis functions in the hidden and output layers. Finally, the last challenging problem is the prediction of energy flux from waves and tides. For this, a multitask EANN has been proposed aiming to predict the energy flux at several prediction time horizons (from 6h to 48h). All these proposals and techniques have been corroborated and discussed according to physical and atmospheric models. The work developed in this Thesis is supported by 11 JCR-indexed papers in international journals (7 Q1, 3 Q2, 1 Q3), 11 papers in international conferences, and 4 papers in national conferences

Facial expression recognition and intensity estimation.

Doctoral Degree. University of KwaZulu-Natal, Durban.Facial Expression is one of the profound non-verbal channels through which human emotion state is inferred from the deformation or movement of face components when facial muscles are activated. Facial Expression Recognition (FER) is one of the relevant research fields in Computer Vision (CV) and Human-Computer Interraction (HCI). Its application is not limited to: robotics, game, medical, education, security and marketing. FER consists of a wealth of information. Categorising the information into primary emotion states only limit its performance. This thesis considers investigating an approach that simultaneously predicts the emotional state of facial expression images and the corresponding degree of intensity. The task also extends to resolving FER ambiguous nature and annotation inconsistencies with a label distribution learning method that considers correlation among data. We first proposed a multi-label approach for FER and its intensity estimation using advanced machine learning techniques. According to our findings, this approach has not been considered for emotion and intensity estimation in the field before. The approach used problem transformation to present FER as a multilabel task, such that every facial expression image has unique emotion information alongside the corresponding degree of intensity at which the emotion is displayed. A Convolutional Neural Network (CNN) with a sigmoid function at the final layer is the classifier for the model. The model termed ML-CNN (Multilabel Convolutional Neural Network) successfully achieve concurrent prediction of emotion and intensity estimation. ML-CNN prediction is challenged with overfitting and intraclass and interclass variations. We employ Visual Geometric Graphics-16 (VGG-16) pretrained network to resolve the overfitting challenge and the aggregation of island loss and binary cross-entropy loss to minimise the effect of intraclass and interclass variations. The enhanced ML-CNN model shows promising results and outstanding performance than other standard multilabel algorithms. Finally, we approach data annotation inconsistency and ambiguity in FER data using isomap manifold learning with Graph Convolutional Networks (GCN). The GCN uses the distance along the isomap manifold as the edge weight, which appropriately models the similarity between adjacent nodes for emotion predictions. The proposed method produces a promising result in comparison with the state-of-the-art methods.Author's List of Publication is on page xi of this thesis

Comb size, shape complexity and laterality of laying hens reared in environments varying in resource choice

The comb is an ornament involved in signalling condition in domestic fowl. We hypothesised that comb size, comb shape complexity (i.e., rugosity, the comb perimeter jaggedness), and comb laterality of laying hens would be influenced by the degree of environmental enrichment experienced during juvenile development in the form of resource choice. We conducted a 2 x 2 factorial crossover experiment with pullets reared in pens containing four perches of equal length and four litter areas of equal size. Pullets were exposed to a single choice vs multiple choices of perch and litter types (i.e., all the same vs all different) during Weeks 1-4 (Period 1) and/or Weeks 5-15 (Period 2) of rearing (n = 4 pens/treatment combination) prior to transfer to standard adult laying pens for Weeks 16-27 (Period 3). In Week 27, combs were photographed, and comb laterality (hanging on left or right side) was noted. Using a custom-made image analysis programme, we captured comb area (mm 2 ), perimeter length (mm), and rugosity ((perimeter length - horizontal length) / horizontal length) from comb photographs of 6-7 randomly selected hens/pen. We predicted that hens reared in the multi -choice environment during Periods 1 and 2 would have larger, more complex, and left -side -biased combs than those in the other treatment groups, reflecting lower allostatic load. The predicted comb side bias was based on a possible bias in head posture/movements associated with greater right eye/ear use and left -brain hemispheric dominance. Contrary to our predictions, we detected an overall right -side bias in comb laterality, and no associations between resource choice treatment in Period 1 or Period 2 and comb area, perimeter length, rugosity, or laterality of the adult hens. Thus, variation in allostatic load resulting from the rearing treatments was insufficient to modify the trajectory of comb morphological development, possibly due to a ceiling effect when comparing environmental treatments on the positive end of the welfare spectrum. We found that left -lopping combs had shorter perimeters than right -lopping combs. However, among hens with left -lopping combs, those with larger combs were heavier and had less feather damage, while among hens with right -lopping combs, those with longer -perimeter combs were heavier and tended to have less comb damage. In conclusion, comb characteristics were related to physical condition at the individual level but did not serve as sensitive integrated indicators of hen welfare in response to basic vs enhanced resource choice during rearing. (c) 2024 The Author(s). Published by Elsevier B.V. on behalf of The Animal Consortium. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/)

Ordinal Shape Coding and Correlation for Orientation-invariant 2D Shape Matching

The human brain and visual system is highly robust and efficient at recognising objects. Although biologically inspired approaches within the field of Computer Vision are often considered as state of the art, a complete understanding of how the brain and visual system works has not yet been unlocked. Benefits of such an understanding are twofold with respect to Computer Vision: firstly, a more robust object recognition system could be produced and secondly a computer architecture as efficient as the brain and visual system would significantly reduce power requirements. Therefore it is worthy to pursue and evaluate biologically inspired theories of object recognition. This engineering doctorate thesis provides an implementation and evaluation of a biologically inspired theory of object recognition called Ordinal Shape Coding and Correlation (OSCC). The theory is underpinned by relative coding and correlation within the human brain and visual system. A derivation of the theory is illustrated with respect to an implementation alongside proposed extensions. As a result, a hierarchical sequence alignment method is proposed for the correlation of multi- dimensional ordinal shape descriptors for the context of orientation-invariant 2D shape descriptor matching. Orientation-invariant 2D shape descriptor matching evaluations are presented which cover both synthetic data and the public MNIST handwritten digits dataset. Synthetic data evaluations show that the proposed OSCC method can be used as a discriminative orientation-invariant 2D shape descriptor. Furthermore, it is shown that the close competitor Shape Context (SC) method outperforms the OSCC method when applied to the MNIST handwritten digits dataset. However, it is shown that OSCC outperforms the SC method when appearance and bending energy costs are removed from the SC method to compare pure shape descriptors. Future work proposes that bending energy and appearance costs are integrated into the OSCC pipeline for further OCR evaluations