Integrating 3D Objects and Pose Estimation for Multimodal Video Annotations

With the recent technological advancements, using video has become a focal point on many ubiquitous activities, from presenting ideas to our peers to studying specific events or even simply storing relevant video clips. As a result, taking or making notes can become an invaluable tool in this process by helping us to retain knowledge, document information, or simply reason about recorded contents. This thesis introduces new features for a pre-existing Web-Based multimodal anno- tation tool, namely the integration of 3D components in the current system and pose estimation algorithms aimed at the moving elements in the multimedia content. There- fore, the 3D developments will allow the user to experience a more immersive interaction with the tool by being able to visualize 3D objects either in a neutral or 360º background to then use them as traditional annotations. Afterwards, mechanisms for successfully integrating these 3D models on the currently loaded video will be explored, along with a detailed overview of the use of keypoints (pose estimation) to highlight details in this same setting. The goal of this thesis will thus be the development and evaluation of these features seeking the construction of a virtual environment in which a user can successfully work on a video by combining different types of annotations.Ao longo dos anos, a utilização de video tornou-se um aspecto fundamental em várias das atividades realizadas no quotidiano como seja em demonstrações e apresentações profissionais, para a análise minuciosa de detalhes visuais ou até simplesmente para preservar videos considerados relevantes. Deste modo, o uso de anotações no decorrer destes processos e semelhantes, constitui um fator de elevada importância ao melhorar potencialmente a nossa compreensão relativa aos conteúdos em causa e também a ajudar a reter características importantes ou a documentar informação pertinente. Efetivamente, nesta tese pretende-se introduzir novas funcionalidades para uma fer- ramenta de anotação multimodal, nomeadamente, a integração de componentes 3D no sistema atual e algorítmos de Pose Estimation com vista à deteção de elementos em mo- vimento em video. Assim, com estas features procura-se proporcionar um experiência mais imersiva ao utilizador ao permitir, por exemplo, a visualização preliminar de objec- tos num plano tridimensional em fundos neutros ou até 360º antes de os utilizar como elementos de anotação tradicionais. Com efeito, serão explorados mecanismos para a integração eficiente destes modelos 3D em video juntamente com o uso de keypoints (pose estimation) permitindo acentuar pormenores neste ambiente de visualização. O objetivo desta tese será, assim, o desenvol- vimento e avaliação continuada destas funcionalidades de modo a potenciar o seu uso em ambientes virtuais em simultaneo com as diferentes tipos de anotações já existentes

Multi-sensor human action recognition with particular application to tennis event-based indexing

The ability to automatically classify human actions and activities using vi- sual sensors or by analysing body worn sensor data has been an active re- search area for many years. Only recently with advancements in both fields and the ubiquitous nature of low cost sensors in our everyday lives has auto- matic human action recognition become a reality. While traditional sports coaching systems rely on manual indexing of events from a single modality, such as visual or inertial sensors, this thesis investigates the possibility of cap- turing and automatically indexing events from multimodal sensor streams. In this work, we detail a novel approach to infer human actions by fusing multimodal sensors to improve recognition accuracy. State of the art visual action recognition approaches are also investigated. Firstly we apply these action recognition detectors to basic human actions in a non-sporting con- text. We then perform action recognition to infer tennis events in a tennis court instrumented with cameras and inertial sensing infrastructure. The system proposed in this thesis can use either visual or inertial sensors to au- tomatically recognise the main tennis events during play. A complete event retrieval system is also presented to allow coaches to build advanced queries, which existing sports coaching solutions cannot facilitate, without an inordi- nate amount of manual indexing. The event retrieval interface is evaluated against a leading commercial sports coaching tool in terms of both usability and efficiency

Design of a Controlled Language for Critical Infrastructures Protection

We describe a project for the construction of controlled language for critical infrastructures protection (CIP). This project originates from the need to coordinate and categorize the communications on CIP at the European level. These communications can be physically represented by official documents, reports on incidents, informal communications and plain e-mail. We explore the application of traditional library science tools for the construction of controlled languages in order to achieve our goal. Our starting point is an analogous work done during the sixties in the field of nuclear science known as the Euratom Thesaurus.JRC.G.6-Security technology assessmen

Use of IoT technologies to improve shooting performance in basketball

Dissertation presented as the partial requirement for obtaining a Master's degree in Information Management, specialization in Knowledge Management and Business IntelligenceTechnology is revolutionizing the world of sports in every way, from the experience of fans to the making and customising of training plans and even refereeing. Inertial sensors are now being used in many sports as they allow effective tracking of metrics that were previously not “within reach” without affecting the performance of players, due to the improvement of their size and “durability”. But it is not just the technological component that is evolving; new strategies and tactical displays are being increasingly seen in several sports, such as basketball. Indeed, in the NBA, in recent years a new trend has emerged as teams are shooting more 3pt shots, and the centre position is progressing as taller players are asked to be more skilled than ever. However, although the game of basketball is changing, a gain in efficiency is not being observed in jump shooting since the percentages of 3pt shots made in the NBA are not increasing as it would be expected from the increase in 3pt shot attempts. The purpose of this study was thus to analyse and make recommendations concerning the use of current technology for tracking shooting performance, as well as the use of new sensors. In order to do so, the main factors behind shooting success were taken into account, to guarantee that the recommendations were as well-founded as possible. The chosen methodology was design science research, where the proposed artifacts were submitted to validation through interviews, and according to received feedback, the proposed artifacts were updated

Intelligent Sensors for Human Motion Analysis

The book, "Intelligent Sensors for Human Motion Analysis," contains 17 articles published in the Special Issue of the Sensors journal. These articles deal with many aspects related to the analysis of human movement. New techniques and methods for pose estimation, gait recognition, and fall detection have been proposed and verified. Some of them will trigger further research, and some may become the backbone of commercial systems

Differences in swimming stroke mechanics and kinematics derived from tri-axial accelerometers during a 200-IM event in South African national swimmers

Context: Swimming is a highly competitive sport, with elite swimmers and coaches constantly looking for ways to improve and challenge themselves to meet new performance goals. The implementation of technology in swimming has proven to be a vital tool in athlete monitoring and in providing coaches with additional information on the swimmer’s performance. Example of this technology is the use of inertial sensory devices such as tri-axial accelerometers. The accelerometers can be used to provide kinematic information with regards to the swimmer’s stroke rate, stroke length and stroke mechanics. In a typical training session, coaches would have to manually time and count their swimmer’s strokes to be able to gain the kinematic information they require. Hence, the use ofinertial sensory technology, such as accelerometers, would provide the necessary information coaches require, allowing them to concentrate on other performance aspects such as theirswimmer’s technique.Aim and objectives: The aim of this study was to determine the kinematic parameters and swimming stroke mechanics that could be derived from tri-axial accelerometers, during a 200-m individual medley (IM) event in South African national level swimmers. Three objectives were set to meet the aim of the study. The first was to identify and differentiate each of the stroking styles using tri-axial accelerometers. The second was to identify and differentiate the kinematic parametersand stroke mechanicsfor all four strokes using tri-axial accelerometers. The third objective was to implement machine learning to automate the identification and interpretation of the accelerometer data. Method:A quantitative, non-experimental descriptive one group post-test only design was used, in which 15 national level swimmers, of which seven male and eight female (mean ±SD: age: 20.9 ± 2.90 years; height: 173.28 ± 10.61 cm; weight: 67.81 ± 8.09 kg; arm span: 178.21 ± 12.15 cm) were tested. Three anthropometric measures were taken (height, weight and arm span) prior to testing, with two tri-axial accelerometers and Polar V800watch and heart rate belt attached to the swimmers left wrist, upper-back and chest, respectively. All swimmerswere required to perform three main swimming sets: 50-m IM, 100-m variation and 200-mIM. Variousdescriptivestatisticsincluding mean, standard deviation and confidence intervals (95%)were used to describe the data. with further inferential statistics including paired t-test, intra-class correlation and Bland Altman analysis wereused to describe the relationship ivbetween the accelerometer and the manually estimated parameters. Additionally, arepeated measures one-way ANOVA (with post-hoc Tukey HSD test) werealso used in an inter-comparison of the stroke parameters between each of the stroking styles. A confusion matrix wasused to measure the classification accuracy of the machine learning model implemented on the accelerometer data.Results:The accelerometers proved successful in identifyingand discerningthe stroke mechanics for each of the four stroking styles, with the use of video footage to validatethe findings. In the stroke kinematic differentiation, theBland Altman analysisresultsshowed an agreement between themanual method and accelerometer-derived estimates, although a discrepancy was evident for several of the kinematic parameters, with a significant difference found with the estimated lap time, average swimming velocity and stroke rate (paired t-test: p 0.05for all strokes)andbetween freestyle and backstroke for the average stroke rate and stroke length (Tukey:p = 0.0968 andp = 0.997, respectively).Lastly, the machine learning model found a classification accuracy of 96.6% in identifyingand labelling the stroking styles fromthe accelerometer data.Conclusion: It was shown that the tri-axial accelerometers were successful in the identification and differentiation of all the stroking styles, stroke mechanics and kinematics, although a discrepancy was found with the average swimming velocity, stroke rate and lap time estimations. The machine learning model implemented proved the benefits of using artificial intelligence to ease the data process and interpretation by automatically labelling the accelerometer data. Therefore, the use of tri-axial accelerometers as a coaching aid has major potential in the swimming community. However, further research is required to eliminate the time-consuming data processingand to increasetheaccuracy of the accelerometer in the measurement of all the stroke kinematics