Study of Camera Spectral Reflectance Reconstruction Performance using CPU and GPU Artificial Neural Network Modelling

Reconstruction of reflectance spectra from camera RGB values is possible, if characteristics of the illumination source, optics and sensors are known. If not, additional information about these has to be somehow acquired. If alongside with pictures taken, RGB values of some colour patches with known reflectance spectra are obtained under the same illumination conditions, the reflectance reconstruction models can be created based on artificial neural networks (ANN). In Matlab, multilayer feedforward networks can be trained using different algorithms. In our study we hypothesized that the scaled conjugate gradient back propagation (BP) algorithm when executed on Graphics Processing Unit, is very fast, but in terms of convergence and performance, it does not match Levenberg-Marquardt algorithm (LM), which, on the other hand, executes only on CPU and is therefore much more time-consuming. We also presumed that there exists a correlation between the two algorithms and is manifested through a dependency of MSE to the number of hidden layer neurons, and therefore the faster BP algorithm could be used to narrow the search span with the LM algorithm to find the best ANN for reflectance reconstruction. The conducted experiment confirmed speed superiority of the BP algorithm but also confirmed better convergence and accuracy of reflectance reconstruction with the LM algorithm. The correlation of reflectance recovery results with ANNs modelled by both training algorithms was confirmed, and a strong correlation was found between the 3rd order polynomial approximation of the LM and BP algorithm\u27s test performances for both mean and best performance

Use of artificial neural networks for the reconstruction of spectral values of color images

Barvo objektov najpogosteje opišemo s slikami, zajetimi s komercialno RGB kamero. Tak opis je odvisen od lastnosti naprave in osvetlitve objekta. Od teh dejavnikov je neodvisen opis objekta s spektrom refleksije, katerega zajem za točkovne odčitke omogoča spektrofotometer, in za večje objekte multispektralna ali hiperspektralna kamera. Te so drage, kar spodbuja raziskave glede možnosti preslikave posnetkov RGB kamere v spekter refleksije. Predlaganih je bilo mnogo metod, od popolnoma matematičnih modelov do pristopov z modeliranjem umetnih nevronskih omrežij (UNO) različnih arhitektur in kompleksnosti. Večina pristopov za modeliranje potrebuje podatke o lastnostih kamere in osvetlitvi. Pri metodi z uporabo UNO, predstavljeni v naši raziskavi, ki temelji na enostavnem polno povezanem nevronskem omrežju z nelinearnimi aktivacijskimi funkcijami nevronov v skritih plasteh, ter manjšim številom vhodov in večjim številom izhodov za modeliranje ne potrebujemo znanja o značilnostih kamere, njenih tipal in osvetlitvi, saj podatke za učenje UNO pridobimo s hkratnim zajemom objekta in referenčnih vzorcev. Posebna pozornost je bila usmerjena v ugotavljanje vplivov hiperparametrov na uspešnost rekonstrukcije spektra refleksije z modeli UNO glede na izbor učnega algoritma, velikost učne množice, število vhodnih podatkov – RGB odčitkov oz. število kamer, število nevronov v skritih plasteh in število skritih plasti, pri čemer smo postavili pet izhodiščnih delovnih hipotez in njihovo pravilnost raziskali v treh korakih, opisanih v člankih revij z indeksom SCI. Verjetnost iskanja uspešnih modelov UNO se z več iteracijami modeliranja pri izbrani konfiguraciji povečuje, a iskanje uspešnejših modelov je časovno zahtevno. Predlagana sta bila pristopa za učinkovitejše iskanje modelov UNO – prvi postopek s hitrejšim, a manj učinkovitim algoritmom učenja enoplastnih UNO, ki se izvaja na grafičnem procesorju, zoži območje iskanja za drugi, počasnejši, a učinkovitejši algoritem, ki se izvaja na centralni procesni enoti, in drugi postopek, ki glede na izbor hiperparametrov modelov UNO in izbor kriterijske funkcije predlaga center iskanja – število nevronov v skritih plasteh –, okoli katerega v ožji okolici iščemo najučinkovitejše modele UNO.The colour of objects is most often described using images captured with a commercial RGB camera. Such a description depends on the characteristics of the capturing device and the object illumination. Independent of these factors is the object description using the reflectance spectrum, which can be captured by a spectrophotometer for point readings and by a multi- or hyper-spectral camera for larger objects. The latter are expensive, which stimulates research into the possibility of mapping RGB camera images to the reflectance spectrum. Many methods have been proposed, ranging from purely mathematical models to artificial neural network (ANN) modelling approaches of different architectures and complexity. Most modelling approaches need information on camera properties and illumination. In the ANN method presented in our study, which is based on a simple, fully connected neural network with nonlinear activation functions of neurons in hidden layers and a smaller number of inputs and a larger number of outputs, the modelling does not require knowledge of the camera characteristics, its sensors and illumination, as the data for ANN learning is obtained by simultaneously capturing the object and the reference samples. Special attention was paid to determine the influence of hyperparameters on the performance of reflectance spectrum reconstruction using ANN models with respect to the choice of the learning algorithm, the size of the training set, the number of input data – RGB readings or the number of cameras, the number of neurons in the hidden layers, and the number of hidden layers, setting five initial working hypotheses and investigating their validity in a three-step study, described in SCI-indexed journal articles. The probability of finding successful ANN models increases with more modelling iterations for the chosen configuration, but finding more successful models is time-consuming. Two approaches are proposed to make the search for ANN models more efficient. The first procedure, using a faster but less efficient single-layer ANN learning algorithm executed on a graphics processor, narrows the search area for a second, slower, more efficient algorithm executed on a central processing unit, and the second procedure, which, given a selection of hyperparameters of the ANN models and a selection of a criterion function, proposes a search centre as the number of neurons in the hidden layers, around which to search for the best-performing ANN models in a narrower neighbourhood

3D modeling of the human body for the purposes of virtual reality

U ovom su radu opisane razne tehnologije koje se koriste kod izrade 3D modela. Uz tehnologije, opisana je i moguća primjena 3D modela za potrebe prividne stvarnosti, konkretno za 3D modele ljudskog tijela. U praktičnom dijelu rada prikazan je postupak izrade jednog 3D modela ljudske glave i tijela

3D modeling of the human body for the purposes of virtual reality

U ovom su radu opisane razne tehnologije koje se koriste kod izrade 3D modela. Uz tehnologije, opisana je i moguća primjena 3D modela za potrebe prividne stvarnosti, konkretno za 3D modele ljudskog tijela. U praktičnom dijelu rada prikazan je postupak izrade jednog 3D modela ljudske glave i tijela

3D modeling of the human body for the purposes of virtual reality

U ovom su radu opisane razne tehnologije koje se koriste kod izrade 3D modela. Uz tehnologije, opisana je i moguća primjena 3D modela za potrebe prividne stvarnosti, konkretno za 3D modele ljudskog tijela. U praktičnom dijelu rada prikazan je postupak izrade jednog 3D modela ljudske glave i tijela

Comparison of Artificial Neural Network and Polynomial Approximation Models for Reflectance Spectra Reconstruction

Knowledge of surface reflection of an object is essential in many technological fields, including graphics and cultural heritage. Compared to direct multi- or hyper-spectral capturing approaches, commercial RGB cameras allow for a high resolution and fast acquisition, so the idea of mapping this information into a reflectance spectrum (RS) is promising. This study compared two modelling approaches based on a training set of RGB-reflectance pairs, one implementing artificial neural networks (ANN) and the other one using multivariate polynomial approximation (PA). The effect of various parameters was investigated: the ANN learning algorithm—standard backpropagation (BP) or Levenberg-Marquardt (LM), the number of hidden layers (HLs) and neurons, the degree of multivariate polynomials in PA, the number of inputs, and the training set size on both models. In the two-layer ANN with significantly fewer inputs than outputs, a better MSE performance was found where the number of neurons in the first HL was smaller than in the second one. For ANNs with one and two HLs with the same number of neurons in the first layer, the RS reconstruction performance depends on the choice of BP or LM learning algorithm. RS reconstruction methods based on ANN and PA are comparable, but the ANN models’ better fine-tuning capabilities enable, under realistic constraints, finding ANNs that outperform PA models. A profiling approach was proposed to determine the initial number of neurons in HLs—the search centre of ANN models for different training set sizes

Exploiting Nonlinearity between Parallel Channels of Multiple Cameras for Accurate ANN Reconstruction of Reflectance Spectra

Colour of an observed object is unambiguously described by its reflectance. Translation from a colour description in RGB space obtained with a digital camera into reflectance, independent of illuminant and camera\u27s sensor characteristics, was performed through an artificial neural network (ANN). In the study, it was hypothesized that the ANN\u27s performance of reflectance reconstruction could be improved by using extended learning datasets with two or three cameras RGB input sets instead of one, but only if the parallel channels of cameras used are not linearly dependent. Nonlinearity was assessed by a quantitative measure of nonlinearity (QMoN), the metric primarily developed for use in chemistry. A noticeable reflectance performance improvement has been found with two and three cameras, even though the cameras\u27 parallel channels exerted only small degrees of nonlinearity. Close attention was paid to the impact of scattering of RGB readings around the ideal values on the reflectance reconstruction performance, and it has been found that the more pronounced scattering is inversely proportional to the performance of ANNs trained with a single-camera input learning set but shows no visible impact on the performance of ANNs trained with extended input learning sets