Application of augmented reality and robotic technology in broadcasting: A survey

As an innovation technique, Augmented Reality (AR) has been gradually deployed in the broadcast, videography and cinematography industries. Virtual graphics generated by AR are dynamic and overlap on the surface of the environment so that the original appearance can be greatly enhanced in comparison with traditional broadcasting. In addition, AR enables broadcasters to interact with augmented virtual 3D models on a broadcasting scene in order to enhance the performance of broadcasting. Recently, advanced robotic technologies have been deployed in a camera shooting system to create a robotic cameraman so that the performance of AR broadcasting could be further improved, which is highlighted in the paper

A low-cost, flexible and portable volumetric capturing system

Multi-view capture systems are complex systems to engineer. They require technical knowledge to install and intricate processes to setup related mainly to the sensors’ spatial alignment (i.e. external calibration). However, with the ongoing developments in new production methods, we are now at a position where the production of high quality realistic 3D assets is possible even with commodity sensors. Nonetheless, the capturing systems developed with these methods are heavily intertwined with the methods themselves, relying on custom solutions and seldom - if not at all - publicly available. In light of this, we design, develop and publicly offer a multi-view capture system based on the latest RGB-D sensor technology. For our system, we develop a portable and easy-to-use external calibration method that greatly reduces the effort and knowledge required, as well as simplify the overall process

Robotic Cameraman for Augmented Reality based Broadcast and Demonstration

In recent years, a number of large enterprises have gradually begun to use vari-ous Augmented Reality technologies to prominently improve the audiences’ view oftheir products. Among them, the creation of an immersive virtual interactive scenethrough the projection has received extensive attention, and this technique refers toprojection SAR, which is short for projection spatial augmented reality. However,as the existing projection-SAR systems have immobility and limited working range,they have a huge difficulty to be accepted and used in human daily life. Therefore,this thesis research has proposed a technically feasible optimization scheme so thatit can be practically applied to AR broadcasting and demonstrations. Based on three main techniques required by state-of-art projection SAR applica-tions, this thesis has created a novel mobile projection SAR cameraman for ARbroadcasting and demonstration. Firstly, by combining the CNN scene parsingmodel and multiple contour extractors, the proposed contour extraction pipelinecan always detect the optimal contour information in non-HD or blurred images.This algorithm reduces the dependency on high quality visual sensors and solves theproblems of low contour extraction accuracy in motion blurred images. Secondly, aplane-based visual mapping algorithm is introduced to solve the difficulties of visualmapping in these low-texture scenarios. Finally, a complete process of designing theprojection SAR cameraman robot is introduced. This part has solved three mainproblems in mobile projection-SAR applications: (i) a new method for marking con-tour on projection model is proposed to replace the model rendering process. Bycombining contour features and geometric features, users can identify objects oncolourless model easily. (ii) a camera initial pose estimation method is developedbased on visual tracking algorithms, which can register the start pose of robot to thewhole scene in Unity3D. (iii) a novel data transmission approach is introduced to establishes a link between external robot and the robot in Unity3D simulation work-space. This makes the robotic cameraman can simulate its trajectory in Unity3D simulation work-space and project correct virtual content. Our proposed mobile projection SAR system has made outstanding contributionsto the academic value and practicality of the existing projection SAR technique. Itfirstly solves the problem of limited working range. When the system is running ina large indoor scene, it can follow the user and project dynamic interactive virtualcontent automatically instead of increasing the number of visual sensors. Then,it creates a more immersive experience for audience since it supports the user hasmore body gestures and richer virtual-real interactive plays. Lastly, a mobile systemdoes not require up-front frameworks and cheaper and has provided the public aninnovative choice for indoor broadcasting and exhibitions

Multispectral imaging for preclinical assessment of rheumatoid arthritis models

Rheumatoid arthritis (RA) is a chronic inflammatory autoimmune condition affecting multiple body systems. Murine models of RA are vital in progressing understanding of the disease. The severity of arthritis symptoms is currently assessed in vivo by observations and subjective scoring which are time-consuming and prone to bias and inaccuracy. The main aim of this thesis is to determine whether multispectral imaging of murine arthritis models has the potential to assess the severity of arthritis symptoms in vivo in an objective manner. Given that pathology can influence the optical properties of a tissue, changes may be detectable in the spectral response. Monte Carlo modelling of reflectance and transmittance for varying levels of blood volume fraction, blood oxygen saturation, and water percentage in the mouse paw tissue demonstrated spectral changes consistent with the reported/published physiological markers of arthritis. Subsequent reflectance and transmittance in vivo spectroscopy of the hind paw successfully detected significant spectral differences between normal and arthritic mice. Using a novel non-contact imaging system, multispectral reflectance and transmittance images were simultaneously collected, enabling investigation of arthritis symptoms at different anatomical paw locations. In a blind experiment, Principal Component (PC) analysis of four regions of the paw was successful in identifying all 6 arthritic mice in a total sample of 10. The first PC scores for the TNF dARE arthritis model were found to correlate significantly with bone erosion ratio results from microCT, histology scoring, and the manual scoring method. In a longitudinal study at 5, 7 and 9 weeks the PC scores identified changes in spectral responses at an early stage in arthritis development for the TNF dARE model, before clinical signs were manifest. Comparison of the multispectral image data with the Monte Carlo simulations suggest that in this study decreased oxygen saturation is likely to be the most significant factor differentiating arthritic mice from their normal littermates. The results of the experiments are indicative that multispectral imaging performs well as an assessor of arthritis for RA models and may outperform existing techniques. This has implications for better assessment of preclinical arthritis and hence for better experimental outcomes and improvement of animal welfare

Per-Pixel Calibration for RGB-Depth Natural 3D Reconstruction on GPU

Ever since the Kinect brought low-cost depth cameras into consumer market, great interest has been invigorated into Red-Green-Blue-Depth (RGBD) sensors. Without calibration, a RGBD camera’s horizontal and vertical field of view (FoV) could help generate 3D reconstruction in camera space naturally on graphics processing unit (GPU), which however is badly deformed by the lens distortions and imperfect depth resolution (depth distortion). The camera’s calibration based on a pinhole-camera model and a high-order distortion removal model requires a lot of calculations in the fragment shader. In order to get rid of both the lens distortion and the depth distortion while still be able to do simple calculations in the GPU fragment shader, a novel per-pixel calibration method with look-up table based 3D reconstruction in real-time is proposed, using a rail calibration system. This rail calibration system offers possibilities of collecting infinite calibrating points of dense distributions that can cover all pixels in a sensor, such that not only lens distortions, but depth distortion can also be handled by a per-pixel D to ZW mapping. Instead of utilizing the traditional pinhole camera model, two polynomial mapping models are employed. One is a two-dimensional high-order polynomial mapping from R/C to XW=YW respectively, which handles lens distortions; and the other one is a per-pixel linear mapping from D to ZW, which can handle depth distortion. With only six parameters and three linear equations in the fragment shader, the undistorted 3D world coordinates (XW, YW, ZW) for every single pixel could be generated in real-time. The per-pixel calibration method could be applied universally on any RGBD cameras. With the alignment of RGB values using a pinhole camera matrix, it could even work on a combination of a random Depth sensor and a random RGB sensor

Prediction of milk yield using visual images of cows through deep learning.

Masters Degree. University of KwaZulu-Natal, Pietermaritzburg.The broad objective of the study was to determine, through deep learning, the predictability of milk yield from a cow's image data. The data size of 1238 image pairs (the side-view images and the rear-view images) from 743 Holstein cows within their first or second parity and the cows’ corresponding first lactation 305-day milk yield values were used to train a deep learning model. The data was first split into the training and testing data at the ratio of 80:20, respectively. The training data was then augmented four times more, then again split into training and validation data at the ratio of 80:20, respectively. Three principal analyses were done, i.e. the prediction of milk yield using rear-view images only, the prediction of milk yield using the side-view images only and the prediction of milk yield using a merge of the side-view and rear-view images (the combined-view images). In all three analyses, poor predictions were observed, i.e. R2 values of 0.32 for the model using the side-view image, 0.30 for the model using the rear-view images and 0.38 for the model using combined side and rear images. The mean absolute errors were 1146.4 kg, 1148.3 kg and 1112.9 kg for the side-view, the rear-view and the combined-view models, respectively. The root mean square error values were 1460.7 kg, 1480.5 kg and 1401.2 kg and the mean absolute error percentages were 17.6, 17.3 and 17.0 % for the side-view, rear-view and combined-view models, respectively. Hypotheses tests were also done to check whether there was any difference between these three prediction models. There was no significant difference in performance between all the prediction models (p>0.05), i.e. the side-view model, the rear-view model and the combinedview model. It was concluded that predicting 305-day milk yield of Holstein cows using either view has the same level of accuracy and no additional benefits are derived from using both the rear and the side views. Keywords: Computer vision; deep learning; linear conformation traits; 305-day milk yield; side-view images; rear-view images; combined-view images; Holstein cows

Synchronized Illumination Modulation for Digital Video Compositing

Informationsaustausch ist eines der Grundbedürfnisse der Menschen. Während früher dazu Wandmalereien,Handschrift, Buchdruck und Malerei eingesetzt wurden, begann man später, Bildfolgen zu erstellen, die als sogenanntes ”Daumenkino” den Eindruck einer Animation vermitteln. Diese wurden schnell durch den Einsatz rotierender Bildscheiben, auf denen mit Hilfe von Schlitzblenden, Spiegeln oder Optiken eine Animation sichtbar wurde, automatisiert – mit sogenannten Phenakistiskopen,Zoetropen oder Praxinoskopen. Mit der Erfindung der Fotografie begannen in der zweiten Hälfte des 19. Jahrhunderts die ersten Wissenschaftler wie Eadweard Muybridge, Etienne-Jules Marey und Ottomar Anschütz, Serienbildaufnahmen zu erstellen und diese dann in schneller Abfolge, als Film, abzuspielen. Mit dem Beginn der Filmproduktion wurden auch die ersten Versuche unternommen, mit Hilfe dieser neuen Technik spezielle visuelle Effekte zu generieren, um damit die Immersion der Bewegtbildproduktionen weiter zu erhöhen. Während diese Effekte in der analogen Phase der Filmproduktion bis in die achtziger Jahre des 20.Jahrhunderts recht beschränkt und sehr aufwendig mit einem enormen manuellen Arbeitsaufwand erzeugt werden mussten, gewannen sie mit der sich rapide beschleunigenden Entwicklung der Halbleitertechnologie und der daraus resultierenden vereinfachten digitalen Bearbeitung immer mehr an Bedeutung. Die enormen Möglichkeiten, die mit der verlustlosen Nachbearbeitung in Kombination mit fotorealistischen, dreidimensionalen Renderings entstanden, führten dazu, dass nahezu alle heute produzierten Filme eine Vielfalt an digitalen Videokompositionseffekten beinhalten. ...Besides home entertainment and business presentations, video projectors are powerful tools for modulating images spatially as well as temporally. The re-evolving need for stereoscopic displays increases the demand for low-latency projectors and recent advances in LED technology also offer high modulation frequencies. Combining such high-frequency illumination modules with synchronized, fast cameras, makes it possible to develop specialized high-speed illumination systems for visual effects production. In this thesis we present different systems for using spatially as well as temporally modulated illumination in combination with a synchronized camera to simplify the requirements of standard digital video composition techniques for film and television productions and to offer new possibilities for visual effects generation. After an overview of the basic terminology and a summary of related methods, we discuss and give examples of how modulated light can be applied to a scene recording context to enable a variety of effects which cannot be realized using standard methods, such as virtual studio technology or chroma keying. We propose using high-frequency, synchronized illumination which, in addition to providing illumination, is modulated in terms of intensity and wavelength to encode technical information for visual effects generation. This is carried out in such a way that the technical components do not influence the final composite and are also not visible to observers on the film set. Using this approach we present a real-time flash keying system for the generation of perspectively correct augmented composites by projecting imperceptible markers for optical camera tracking. Furthermore, we present a system which enables the generation of various digital video compositing effects outside of completely controlled studio environments, such as virtual studios. A third temporal keying system is presented that aims to overcome the constraints of traditional chroma keying in terms of color spill and color dependency. ..

An integrated approach to whole-body vibration

Obiettivo di questa tesi è la determinazione e quantificazione degli effetti della whole-body vibration al corpo umano, in termini di consumo energetico, tramite un approccio globale e integrato. L’obiettivo è ottenuto considerando il corpo umano come una struttura organica complessa. Allo scopo di comprendere come questo risponda alle vibrazioni verticali, il consumo energetico del corpo umano è stato misurato per mezzo della variazione della temperatura superficiale con tecniche di misurazione a termografia infrarossa. Lo spostamento dei muscoli invece con il sistema di analisi di movimento Vicon MX. Infine, per quanto riguarda il consumo di ossigeno con il sistema telemetrico Cosmed K4. Il primo passo è stato l’istituzione di un protocollo appropriato che soddisfi l’obiettivo di questo studio. Infatti, la mancanza di coerenza nei protocollo di whole-body vibration che si trovano allo stato dell’arte, ha reso essenziale l’istituzione di un apposito protocollo, ed a questo scopo è stata definita la struttura dell’esperimento. Di conseguenza, è stata avviata una serie di prove per esaminare la risposta del corpo umano alle vibrazioni verticali, cambiando la durata e la frequenza della vibrazione, nonché la durata del periodo di riposo. In totale, quattro persone in piedi sono state sottoposte a vibrazioni verticali, in una pedana vibrante, a frequenze da 20 a 50 Hz. Dopo l’instaurazione del protocollo finale, sono stati avviate una serie di prove di laboratorio. In particolare, sono state scelte tre frequenze per le vibrazioni: 20, 30 e 45 Hz. I risultati ottenuti più interessanti di questo studio, riguardano il consumo di ossigeno, la temperatura superficiale e i coefficienti di trasmissibilità dell’accelerazione.The objective of this thesis is to determine and quantify the effects of whole-body vibration to the human body in terms of energy expenditure, by means of a global and integrated approach. This objective is attained by considering the human body as a complex organic structure. In order to understand how it responds to vertical vibrations, the energy expenditure of the human body was measured by means of the variation in superficial temperature with the aid of infrared thermography, the displacement of the muscles with the aid of the Vicon MX motion analysis system and the oxygen uptake with the aid of the Cosmed K4 telemetric system. The establishment of an appropriate protocol which satisfies the aim of this study was the first goal. The lack of consistency in whole-body vibration protocols in the current published studies makes the establishment of an appropriate protocol essential, and in this sense, an experiment setup was implemented. Therefore, a series of experiments was conducted to examine the response of the human body to vertical vibrations, changing the duration and the frequency of vertical vibration, and the duration of rest period. A number of four persons were subjected to vertical vibrations on a vibrating table in a standing position at a frequency ranging from 20 to 50 Hz. After the establishment of the final protocol, a series of laboratory experiments took place. Three different vibration frequencies were chosen: 20, 30 and 45 Hz corresponding to three different tests. The most interesting findings regard the oxygen consumption, the superficial temperature evolution, and the transmissibility coefficients for the acceleration