Collaborative Learning of Fine-grained Visual Data

Problem: Deep learning based vision systems have achieved near human accuracy in recognizing coarse object categories from visual data. But recognizing fine-grained sub-categories remains an open problem. Tasks like fine-grained species recognition poses further challenges: significant background variation compared to subtle difference between objects, high class imbalance due to scarcity of samples for endangered species, cost of domain expert annotations and labeling, etc. Methodology: The existing approaches, like transfer learning, to solve the problem of learning small specialized datasets are still inadequate in case of fine-grained sub-categories. The hypothesis of this work is that collaborative filters should be incorporated into the present learning frameworks to better address these challenges. The intuition comes from the fact that collaborative representation based classifiers have been earlier used for face recognition problems which present similar challenges. Outcomes: Keeping the above hypothesis in mind, the thesis achieves the following objectives: 1) It demonstrates the suitability of collaborative classifiers for fine-grained recognition 2) It expands the state-of-the-art by incorporating automated background suppression into collaborative classification formulation 3) It incorporates the collaborative cost function into supervised learning (deep convolutional network) and unsupervised learning (clustering algorithms) 4) Lastly, during the work several benchmark fine-grained image datasets have been introduced on NZ and Indian butterflies and bird species recognition

Deep learning to find colorectal polyps in colonoscopy: A systematic literature review

Colorectal cancer has a great incidence rate worldwide, but its early detection significantly increases the survival rate. Colonoscopy is the gold standard procedure for diagnosis and removal of colorectal lesions with potential to evolve into cancer and computer-aided detection systems can help gastroenterologists to increase the adenoma detection rate, one of the main indicators for colonoscopy quality and predictor for colorectal cancer prevention. The recent success of deep learning approaches in computer vision has also reached this field and has boosted the number of proposed methods for polyp detection, localization and segmentation. Through a systematic search, 35 works have been retrieved. The current systematic review provides an analysis of these methods, stating advantages and disadvantages for the different categories used; comments seven publicly available datasets of colonoscopy images; analyses the metrics used for reporting and identifies future challenges and recommendations. Convolutional neural networks are the most used architecture together with an important presence of data augmentation strategies, mainly based on image transformations and the use of patches. End-to-end methods are preferred over hybrid methods, with a rising tendency. As for detection and localization tasks, the most used metric for reporting is the recall, while Intersection over Union is highly used in segmentation. One of the major concerns is the difficulty for a fair comparison and reproducibility of methods. Even despite the organization of challenges, there is still a need for a common validation framework based on a large, annotated and publicly available database, which also includes the most convenient metrics to report results. Finally, it is also important to highlight that efforts should be focused in the future on proving the clinical value of the deep learning based methods, by increasing the adenoma detection rate.This work was partially supported by PICCOLO project. This project has received funding from the European Union's Horizon2020 Research and Innovation Programme under grant agreement No. 732111. The sole responsibility of this publication lies with the author. The European Union is not responsible for any use that may be made of the information contained therein. The authors would also like to thank Dr. Federico Soria for his support on this manuscript and Dr. José Carlos Marín, from Hospital 12 de Octubre, and Dr. Ángel Calderón and Dr. Francisco Polo, from Hospital de Basurto, for the images in Fig. 4

Innovative mathematical and numerical models for studying the deformation of shells during industrial forming processes with the Finite Element Method

The doctoral thesis "Innovative mathematical and numerical models for studying the deformation of shells during industrial forming processes with the Finite Element Method" aims to contribute to the development of finite element methods for the analysis of stamping processes, a problematic area with a clear industrial application. To achieve the proposed objectives, the first part of this thesis covers the solid-shell elements. This type of element is attractive for the simulation of forming processes, since any type of three-dimensional constitutive law can be formulated without the need to consider any additional conjecture. Additionally, the contact of both sides can be easily treated. This work first presents the development of a triangular prismatic solid-sheet element, for the analysis of thick and thin sheets with capacity for large deformations. This element is in total Lagrangian formulation, and uses neighboring elements to compute a field of quadratic displacements. In the original formulation, a modified right Cauchy tensor was obtained; however, in this work, the formulation is extended obtaining a modified strain gradient, which allows the concepts of push-forward and pull-back to be used. These concepts provide a mathematically consistent method for the definition of temporary derivatives of tensors and, therefore, can be used, for example, to work with elasto-plasticity. This work continues with the development of the contact formulation used, a methodology found in the bibliography on computational contact mechanics for implicit simulations. This formulation consists of an exact integration of the contact interface using mortar methods, which allows obtaining the most consistent integration possible between the integration domains, as well as the most exact possible solution. The most notable contribution of this work is the consideration of dual augmented Lagrange multipliers as an optimization method. To solve the system of equations, a semi-smooth Newton method is considered, which consists of an active set strategy, also extensible in the case of friction problems. The formulation is functional for both frictionless and friction problems, which is essential for simulating stamping processes. This frictional formulation is framed in traditional friction models, such as Coulomb friction, but the development presented can be extended to any type of friction model. The remaining necessary component for the simulation of industrial processes are the constitutive models. In this work, this is materialized in the formulation of plasticity considered. These constitutive models will be considered plasticity models for large deformations, with an arbitrary combination of creep surfaces and plastic potentials: the so-called non-associative models. To calculate the tangent tensor corresponding to these general laws, numerical implementations based on perturbation methods have been considered. Another fundamental contribution of this work is the development of techniques for adaptive remeshing, of which different approaches will be presented. On the one hand, metric-based techniques, including the level-set and Hessian approaches. These techniques are general-purpose and can be considered in both structural problems and fluid mechanics problems. On the other hand, the SPR error estimation method, more conventional than the previous ones, is presented. In this area, the contribution of this work consists in the estimation of error using the Hessian and SPR techniques for the application to numerical contact problems.La tesis doctoral "Modelos matemáticos y numéricos innovadores para el estudio de la deformación de láminas durante los procesos de conformado industrial por el Método de los Elementos Finitos" pretende contribuir al desarrollo de métodos de elementos finitos para el análisis de procesos de estampado, un área problemática con una clara aplicación industrial. De hecho, este tipo de problemas multidisciplinares requieren el conocimiento de múltiples disciplinas, como la mecánica de medios continuos, la plasticidad, la termodinámica y los problemas de contacto, entre otros. Para alcanzar los objetivos propuestos, la primera parte de esta tesis abarca los elementos de sólido lámina. Este tipo de elemento resulta atractivo para la simulación de procesos de conformado, dado que cualquier tipo de ley constitutiva tridimensional puede ser formulada sin necesidad de considerar ninguna conjetura adicional. Además, este tipo de elementos permite realizar una descripción tridimensional del cuerpo deformable, por tanto, el contacto de ambas caras puede ser tratado fácilmente. Este trabajo presenta en primer lugar el desarrollo de un elemento de sólido-lámina prismático triangular, para el análisis de láminas gruesas y delgadas con capacidad para grandes deformaciones. Este elemento figura en formulación Lagrangiana total, y emplea los elementos vecinos para poder computar un campo de desplazamientos cuadráticos. En la formulación original, se obtenía un tensor de Cauchy derecho modificado (¯C); sin embargo, en este trabajo, la formulación se extiende obteniendo un gradiente de deformación modificado (¯F), que permite emplear los conceptos de push-forward y pull-back. Dichos conceptos proveen de un método matemáticamente consistente para la definición de derivadas temporales de tensores y, por tanto, puede ser usado, por ejemplo, para trabajar con elasto-plasticidad. El elemento se basa en tres modificaciones: (a) una aproximación clásica de deformaciones transversales de corte mixtas impuestas; (b) una aproximación de deformaciones impuestas para las Componentes en el plano tangente de la lámina; y (c) una aproximación de deformaciones impuestas mejoradas en la dirección normal a través del espesor, mediante la consideración de un grado de libertad adicional. Los objetivos son poder utilizar el elemento para la simulación de láminas sin bloquear por cortante, mejorar el comportamiento membranal del elemento en el plano tangente, eliminar el bloqueo por efecto Poisson y poder tratar materiales elasto-plásticos con un flujo plástico incompresible, así como materiales elásticos cuasi-incompresibles o materiales con flujo plástico isocórico. El elemento considera un único punto de Gauss en el plano, mientras que permite considerar un número cualquiera de puntos de integración en su eje, con el objetivo de poder considerar problemas con una significativa no linealidad en cuanto a plasticidad. Este trabajo continúa con el desarrollo de la formulación de contacto empleada, una metodología que se encuentra en la bibliografía sobre la mecánica de contacto computacional para simulaciones implícitas. Dicha formulación consiste en una integración exacta de la interfaz de contacto mediante métodos de mortero, lo que permite obtener la integración más consistente posible entre los dominios de integración, así como la solución más exacta posible. La implementación también considera varios algoritmos de optimización, como la optimización mediante penalización. La contribución más notable de este trabajo es la consideración de multiplicadores de Lagrange aumentados duales como método de optimización. Estos permiten condensar estáticamente el sistema de ecuaciones, lo que permite eliminar los multiplicadores de Lagrange de la resolución y, por lo tanto, permite la consideración de solvers iterativos. Además, la formulación ha sido adecuadamente linealizada, asegurando la convergencia cuadrática del problema. Para resolver el sistema de ecuaciones, se considera un método de Newton semi-smooth, que consiste en una estrategia de set activo, extensible también en el caso de problemas friccionales. La formulación es funcional tanto para problemas sin fricción como para problemas friccionales, lo que es esencial para la simulación de procesos de estampado. Esta formulación friccional se enmarca en los modelos de fricción tradicionales, como la fricción de Coulomb, pero el desarrollo presentado puede extenderse a cualquier tipo de modelo de fricción. Esta formulación de contacto es totalmente compatible con el elemento sólido-lámina introducido en este trabajo. El componente necesario restante para la simulación de procesos industriales son los modelos constitutivos. En este trabajo, esto se ve materializado en la formulación de plasticidad considerada. Estos modelos constitutivos se considerarán modelos de plasticidad para grandes deformaciones, con una combinación arbitraria de superficies de fluencia y potenciales plásticos: los llamados modelos no asociados. Para calcular el tensor tangente correspondiente a estas leyes generales, se han considerado implementaciones numéricas basadas en métodos de perturbación. Otra contribución fundamental de este trabajo es el desarrollo de técnicas para el remallado adaptativo, de las que se presentarán distintos enfoques. Por un lado, las técnicas basadas en métricas, incluyendo los enfoques level-set y Hessiano. Estas técnicas son de propósito general y pueden considerarse tanto en la aplicación de problemas estructurales como en problemas de mecánica de fluidos. Por otro lado, se presenta el método de estimación de errores SPR, más convencional que los anteriores. En este ámbito, la contribución de este trabajo consiste en la estimación de error mediante las técnicas de Hessiano y SPR para la aplicación a problemas de contacto numérico. Con los desarrollos previamente introducidos, estaremos en disposición de introducir los casos de aplicación centrados en el contexto de procesos de estampado. Es relevante destacar que estos ejemplos son comparados con las soluciones de referencia disponibles en la bibliografía como forma de validar los desarrollos presentados hasta este punto. El presente documento está organizado de la siguiente manera. El primer capítulo establece los objetivos y revisa la bibliografía acerca de los temas clave de este trabajo. El segundo capítulo hace una introducción de la mecánica de medios continuos y los conceptos relativos al Método de los Elementos Finitos (MEF), necesarios en los desarrollos que se presentarán en los capítulos siguientes. El tercer capítulo aborda la formulación del elemento sólido-lámina, así como del elemento de lámina sin grados de libertad de rotación que inspira el sólido-lámina desarrollado. Esta parte muestra varios ejemplos académicos que son comúnmente empleados en la bibliografía como problemas de referencia de láminas. El cuarto capítulo presenta la formulación desarrollada para la resolución de problemas de contacto numérico, consistente en una formulación implícita de integración exacta mediante métodos mortero y multiplicadores de Lagrange aumentados duales. Este capítulo incluye, asimismo, varios ejemplos comúnmente encontrados en la bibliografía, que generalmente son considerados para su validación. El quinto capítulo presenta la formulación de plasticidad empleada, incluyendo algunos detalles técnicos desde el punto de vista de la implementación, así como varios ejemplos de validación. El sexto capítulo muestra los algoritmos de remallado adaptativo desarrollados en el contexto de este trabajo, y presenta varios ejemplos, que incluyen no solo casos estructurales, sino también de mecánica de fluidos. El séptimo capítulo encapsula algunos casos de validación y aplicación para procesos de estampado. El capítulo final comprende las conclusiones, así como los trabajos que podrían continuar el presente estudio.Postprint (published version

Text Similarity Between Concepts Extracted from Source Code and Documentation

Context: Constant evolution in software systems often results in its documentation losing sync with the content of the source code. The traceability research field has often helped in the past with the aim to recover links between code and documentation, when the two fell out of sync. Objective: The aim of this paper is to compare the concepts contained within the source code of a system with those extracted from its documentation, in order to detect how similar these two sets are. If vastly different, the difference between the two sets might indicate a considerable ageing of the documentation, and a need to update it. Methods: In this paper we reduce the source code of 50 software systems to a set of key terms, each containing the concepts of one of the systems sampled. At the same time, we reduce the documentation of each system to another set of key terms. We then use four different approaches for set comparison to detect how the sets are similar. Results: Using the well known Jaccard index as the benchmark for the comparisons, we have discovered that the cosine distance has excellent comparative powers, and depending on the pre-training of the machine learning model. In particular, the SpaCy and the FastText embeddings offer up to 80% and 90% similarity scores. Conclusion: For most of the sampled systems, the source code and the documentation tend to contain very similar concepts. Given the accuracy for one pre-trained model (e.g., FastText), it becomes also evident that a few systems show a measurable drift between the concepts contained in the documentation and in the source code.</p

Mining climate data for shire level wheat yield predictions in Western Australia

Climate change and the reduction of available agricultural land are two of the most important factors that affect global food production especially in terms of wheat stores. An ever increasing world population places a huge demand on these resources. Consequently, there is a dire need to optimise food production. Estimations of crop yield for the South West agricultural region of Western Australia have usually been based on statistical analyses by the Department of Agriculture and Food in Western Australia. Their estimations involve a system of crop planting recommendations and yield prediction tools based on crop variety trials. However, many crop failures arise from adherence to these crop recommendations by farmers that were contrary to the reported estimations. Consequently, the Department has sought to investigate new avenues for analyses that improve their estimations and recommendations. This thesis explores a new approach in the way analyses are carried out. This is done through the introduction of new methods of analyses such as data mining and online analytical processing in the strategy. Additionally, this research attempts to provide a better understanding of the effects of both gradual variation parameters such as soil type, and continuous variation parameters such as rainfall and temperature, on the wheat yields. The ultimate aim of the research is to enhance the prediction efficiency of wheat yields. The task was formidable due to the complex and dichotomous mixture of gradual and continuous variability data that required successive information transformations. It necessitated the progressive moulding of the data into useful information, practical knowledge and effective industry practices. Ultimately, this new direction is to improve the crop predictions and to thereby reduce crop failures. The research journey involved data exploration, grappling with the complexity of Geographic Information System (GIS), discovering and learning data compatible software tools, and forging an effective processing method through an iterative cycle of action research experimentation. A series of trials was conducted to determine the combined effects of rainfall and temperature variations on wheat crop yields. These experiments specifically related to the South Western Agricultural region of Western Australia. The study focused on wheat producing shires within the study area. The investigations involved a combination of macro and micro analyses techniques for visual data mining and data mining classification techniques, respectively. The research activities revealed that wheat yield was most dependent upon rainfall and temperature. In addition, it showed that rainfall cyclically affected the temperature and soil type due to the moisture retention of crop growing locations. Results from the regression analyses, showed that the statistical prediction of wheat yields from historical data, may be enhanced by data mining techniques including classification. The main contribution to knowledge as a consequence of this research was the provision of an alternate and supplementary method of wheat crop prediction within the study area. Another contribution was the division of the study area into a GIS surface grid of 100 hectare cells upon which the interpolated data was projected. Furthermore, the proposed framework within this thesis offers other researchers, with similarly structured complex data, the benefits of a general processing pathway to enable them to navigate their own investigations through variegated analytical exploration spaces. In addition, it offers insights and suggestions for future directions in other contextual research explorations

Multiscale modelling of woven and knitted fabric membranes

Light-weight fabric membranes have gained increasing popularity over the past years due to their tailorable structural and material performances. These tailorable properties include stretch forming and deep drawing formability that exhibits excellent stretchability and drapeability properties of textiles and textile composites. Since the inception of computerised numerical control for three-dimensional textile-manufacturing machines, technical textiles paved their way to numerous applications, certainly not limited to; aerospace, biomedical, civil engineering, defence, marine and medical industries. Digital interlooping and digital interlacing technology in additive manufacturing greatly advanced the manufacturing processes of textiles. In this work, we consider two branches of technical fabrics, namely plain-woven and weft-knitted. Multiscale modelling is the tool of choice for homogenising periodic structures and has been used extensively to model and analyse the mechanical behaviour of woven and knitted fabrics. But there is a plethora of literature discussing the demerits of such conventional multiscale modelling. These demerits include higher computational costs, rigid numerical models, ineffcient algorithmic computations and inability to incorporate geometric nonlinearities. We propose a data-driven nonlinear multiscale modelling technique to analyse the complex mechanical behaviour of plain-woven and weft-knitted fabrics with a neat extension to fabric material designing. We show how the integration of statistical learning techniques mitigates the weaknesses of conventional multiscale modelling. Moreover, we discuss the avenues that will open in many potential fields with regard to material modelling, structural engineering and textile industries. In the proposed data-driven nonlinear computational homogenisation technique, we effi ciently integrate the microscale and macroscale using Gaussian Process Regression (GPR) statistical learning technique. In the microscale, representative volume elements (RVEs) are modelled using nite deformable isogeometric spatial rods and deformation is homogenised using periodic boundary conditions. This nite deformable rod is profi cient in handling large deformations, rod-to-rod contacts, arbitrary cross-section de finitions and follower loads. Respecting the principle of separation of scales, we construct response databases by applying different homogenised strain states to the RVEs and recording the respective incremental volume-averaged energy values. We use GPR to learn a model using a 5-fold cross-validation technique by optimising the log marginal likelihood. In the macroscale, textiles are modelled as nonlinear orthotropic membranes for which the stresses and material constitutive relations are predicted by the trained GPR model. This coupling between GPR and membrane models is achieved through a systematic and seamless nite element integration using C++ and Python environments. A neat extension to material designing is also discussed with potentials to extend the work into other related fi elds.Cambridge trust and Trinity Hall scholarshi

Tracking interacting targets in multi-modal sensors

PhDObject tracking is one of the fundamental tasks in various applications such as surveillance, sports, video conferencing and activity recognition. Factors such as occlusions, illumination changes and limited field of observance of the sensor make tracking a challenging task. To overcome these challenges the focus of this thesis is on using multiple modalities such as audio and video for multi-target, multi-modal tracking. Particularly, this thesis presents contributions to four related research topics, namely, pre-processing of input signals to reduce noise, multi-modal tracking, simultaneous detection and tracking, and interaction recognition. To improve the performance of detection algorithms, especially in the presence of noise, this thesis investigate filtering of the input data through spatio-temporal feature analysis as well as through frequency band analysis. The pre-processed data from multiple modalities is then fused within Particle filtering (PF). To further minimise the discrepancy between the real and the estimated positions, we propose a strategy that associates the hypotheses and the measurements with a real target, using a Weighted Probabilistic Data Association (WPDA). Since the filtering involved in the detection process reduces the available information and is inapplicable on low signal-to-noise ratio data, we investigate simultaneous detection and tracking approaches and propose a multi-target track-beforedetect Particle filtering (MT-TBD-PF). The proposed MT-TBD-PF algorithm bypasses the detection step and performs tracking in the raw signal. Finally, we apply the proposed multi-modal tracking to recognise interactions between targets in regions within, as well as outside the cameras’ fields of view. The efficiency of the proposed approaches are demonstrated on large uni-modal, multi-modal and multi-sensor scenarios from real world detections, tracking and event recognition datasets and through participation in evaluation campaigns

Real-Time Hybrid Visual Servoing of a Redundant Manipulator via Deep Reinforcement Learning

Fixtureless assembly may be necessary in some manufacturing tasks and environ-ments due to various constraints but poses challenges for automation due to non-deterministic characteristics not favoured by traditional approaches to industrial au-tomation. Visual servoing methods of robotic control could be effective for sensitive manipulation tasks where the desired end-effector pose can be ascertained via visual cues. Visual data is complex and computationally expensive to process but deep reinforcement learning has shown promise for robotic control in vision-based manipu-lation tasks. However, these methods are rarely used in industry due to the resources and expertise required to develop application-specific systems and prohibitive train-ing costs. Training reinforcement learning models in simulated environments offers a number of benefits for the development of robust robotic control algorithms by reducing training time and costs, and providing repeatable benchmarks for which algorithms can be tested, developed and eventually deployed on real robotic control environments. In this work, we present a new simulated reinforcement learning envi-ronment for developing accurate robotic manipulation control systems in fixtureless environments. Our environment incorporates a contemporary collaborative industrial robot, the KUKA LBR iiwa, with the goal of positioning its end effector in a generic fixtureless environment based on a visual cue. Observational inputs are comprised of the robotic joint positions and velocities, as well as two cameras, whose positioning reflect hybrid visual servoing with one camera attached to the robotic end-effector, and another observing the workspace respectively. We propose a state-of-the-art deep reinforcement learning approach to solving the task environment and make prelimi-nary assessments of the efficacy of this approach to hybrid visual servoing methods for the defined problem environment. We also conduct a series of experiments ex-ploring the hyperparameter space in the proposed reinforcement learning method. Although we could not prove the efficacy of a deep reinforcement approach to solving the task environment with our initial results, we remain confident that such an ap-proach could be feasible to solving this industrial manufacturing challenge and that our contributions in this work in terms of the novel software provide a good basis for the exploration of reinforcement learning approaches to hybrid visual servoing in accurate manufacturing contexts