Deep Bilateral Learning for Real-Time Image Enhancement

Performance is a critical challenge in mobile image processing. Given a reference imaging pipeline, or even human-adjusted pairs of images, we seek to reproduce the enhancements and enable real-time evaluation. For this, we introduce a new neural network architecture inspired by bilateral grid processing and local affine color transforms. Using pairs of input/output images, we train a convolutional neural network to predict the coefficients of a locally-affine model in bilateral space. Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation. At runtime, the neural network consumes a low-resolution version of the input image, produces a set of affine transformations in bilateral space, upsamples those transformations in an edge-preserving fashion using a new slicing node, and then applies those upsampled transformations to the full-resolution image. Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators. Unlike previous work, our model is trained off-line from data and therefore does not require access to the original operator at runtime. This allows our model to learn complex, scene-dependent transformations for which no reference implementation is available, such as the photographic edits of a human retoucher.Comment: 12 pages, 14 figures, Siggraph 201

Applicability of low-cost cameras for monitoring suspended sediment in rivers through close-range remote sensing

Suspended sediment in rivers is a major problem globally. Monitoring of water turbidity and suspended sediment concentration (SSC) using satellites and in-situ sampling has been used widely to assess fine sediment pollution. However, due to low image resolution, application of satellite remote sensing is limited to only large water bodies, while in-situ sampling does not provide the continuous spatial data that are needed to address certain scientific questions or management problems. This research aimed to understand the potential of using low-cost cameras to estimate SSC in smaller rivers and streams and produce reach scale ‘maps’ of SSC. The study consists of development and testing of statistical models to predict SSC from pixel information contained in digital images, and validation of these models through field tests. An overarching goal was to assess the transferability of models between rivers and the effects of different camera sensors on SSC predictions. Laboratory experiments developed predictive models for two cameras (Vivo V9 smartphone and DJI Mavic Pro drone). Experiments involved manipulation of SSC in a water filled tank, with images taken with each camera and over a different coloured bed at each controlled sediment concentration. Digital Number (DN) values for each bed colour, camera and colour channel combination was extracted, with Generalised Additive Models fitted to Red, Blue and Green (R, G, B) colour bands. In general, there were significant relations between SSC and the mean DN values, with G and B most frequently providing the best fits. Relations differed appreciably depending on bed characteristics, as a function of the relative colour of the bed and the material in suspension; some relations were direct (positive) and some indirect (negative). Thus, laboratory tests indicated that predictive relations need to be developed on a river-by-river basis due to differences in bed characteristics. There were some subtle differences between the two cameras, but in general both yielded images from which SSC could be predicted reliably in laboratory conditions. However, almost all relations broke down at very high SSCs depending on the bed colour, camera and colour channel combination; once the amount of fine material in suspension exceeded a certain threshold, SSC could not be predicted reliably from DN values. The field tests demonstrated that it is possible to produce accurate maps of SSC using an orthomosaic developed directly using DN values. These involved developing a calibration relationship for SSC v DN from images collected from drone flights at 30 m height above a reach of the Semenyih River, Malaysia. This relationship successfully predicted SSC, with the B colour band providing the best fit (R2 >0.86 for the observed v predicted). The SSC map was able to shed light on the influence of a tributary on main stem SSCs and patterns of mixing of the fine sediment delivered by the tributary. Such fine scale spatial patterns (1cm2/pixel) are evident neither from satellite data nor in-situ monitoring. The methods presented here are applicable to a variety of questions and contexts, from understanding downstream changes in SSC in glacial rivers to assessing effects of forest loss on SSC in tropical systems

High Quality Image Reconstruction from RAW and JPEG Image Pair

A camera RAW file contains minimally processed data from the image sensor. The contents of the RAW file include more information, and potentially higher quality, than the commonly used JPEG file. But the RAW file is typically several times larger than the JPEG file (taking fewer images, slower quick shooting) and lacks the standard file format (not ready-to-use, prolonging the image workflow). These drawbacks limit its applications. In this paper, we suggest a new “hybrid ” image capture mode: a high-res JPEG file and a low-res RAW file as alternative of the original RAW file. Most RAW users can be benefited from such a combination. To address this problem, we provide an effective approach to reconstruct a high quality image by combining the advantages of two kinds of files. We formulate this reconstruction process as a global optimization problem by enforcing two constraints: reconstruction constraint and detail consistency constraint. The final recovered image is smaller than the full-res RAW file, enables faster quick shooting, and has both richer information (e.g., color space, dynamic range, lossless 14 bits data) and higher resolution. In practice, the functionality of capturing such a “hybrid ” image pair in one-shot has been supported in some existing digital cameras. 1

Objektivizacija vizualne prihvatljivosti degradacije originala fotografske slike = Objectification of the original photographic image degradation visual acceptability

Fotografska slika je najčešći medij prenošenja informacija dvodimenzionalnom statičnom slikom. Kako bi se osigurao odabir fotografskih slika koje prenose željene informacije, potrebno je osigurati objektivizaciju njihove vizualne prihvatljivosti. Odabir fotografskih slika koje prenose određene informacije, bez obzira ostvaruje li se ta slika kao samostalan medij ili kroz druge, medije, tiskane ili elektronske, primarno se temelji na vizualnim procjenama. Informacije koje prenosi fotografska slika određene su njezinom semantikom koja je definirana tehničkim i sintaktičkim aspektima. U disertaciji se provode istraživanja vizualnim uspoređivanjem kreiranih originala i obradom degradiranih originala fotografskih slika te mjernim određivanjem ukupne promjene boja, svjetline, kromatičnosti, dinamičkog raspona, vrijednosti histograma, sposobnosti razdvajanja linija i faktora sličnosti kao pokazatelja njihovih deskriptivnih karakteristika. Istraživanja upućuju na mogućnost definiranja granica prihvatljivosti promjena pojedinih tehničkih i sintaktičkih vrijednosti fotografske slike uz zadržavanje informacija koje prenosi, kao i mogućnost upravljanja pojedinim parametrima digitalnog zapisa fotografske slike sa svrhom mijenjanja informacije koja se prenosi uz zadržavanje ikoničkog karaktera konkretne fotografske slike odnosno prihvaćanja te slike kao realnog zapisa. U području promjena plavog, zelenog i crvenog kanala digitalnog zapisa fotografske slike u granicama zadržavanja ikoničkog karaktera, istraživanje dokazuje povezanost mjernih karakteristika boja fotografske slike i percepcije fotografske slike, ali i povezanost granica prihvatljivosti promjena vrijednosti kanala zapisa i motiva fotografske slike. Provedena istraživanja povezuju percepciju, odnosno vizualnu procjenu, fotografske slike s vrijednostima promijene dinamičkog raspona, ekspozicije, tonskih vrijednosti, ukupne promijene boja ∆Eₒₒ, svjetline i sposobnosti razdvajanja linija kao parametara objektivne procjene promjena definiranih fotografskih slika u odnosu na originalnu fotografsku sliku, te ukazuju na povezanost navedenih mjernih karakteristika fotografske slike, faktora sličnosti te vizualnih ekspertnih i procjena šire skupine konzumenata. Potvrđuju mogućnost objektivizacije procjene degradacije originala fotografske slike, odnosno prihvatljivosti te degradacije, a da degradirana fotografska slika zadržava ikoničnost u odnosu na originalnu fotografsku sliku