An adaptive browsing-based approach for creating a photographic story

Pictures are often self-explanatory; they capture a moment in time. However, a single photo cannot represent the whole moment. The creation of photographic stories is a means to better preserve memories. Relying on the content-based and contextual metadata within digital photos we could assist users to explore their collection to create the stories with regard to their events and experiences. In this poster we propose a novel approach for supporting the self creation of stories with digital photographs. By incorporating an adaptive learning scheme to capture implicit user feedback, our approach supports content and context assisted browsing

Audeosynth: music-driven video montage

We introduce music-driven video montage, a media format that offers a pleasant way to browse or summarize video clips collected from various occasions, including gatherings and adventures. In music-driven video montage, the music drives the composition of the video content. According to musical movement and beats, video clips are organized to form a montage that visually reflects the experiential properties of the music. Nonetheless, it takes enormous manual work and artistic expertise to create it. In this paper, we develop a framework for automatically generating music-driven video montages. The input is a set of video clips and a piece of background music. By analyzing the music and video content, our system extracts carefully designed temporal features from the input, and casts the synthesis problem as an optimization and solves the parameters through Markov Chain Monte Carlo sampling. The output is a video montage whose visual activities are cut and synchronized with the rhythm of the music, rendering a symphony of audio-visual resonance.postprin

Best Photo Selection

The rising of digital photography underlies a clear change on the paradigm of the photography management process by amateur photographers. Nowadays, taking one more photo comes for free, thus it is usual for amateurs to take several photos of the same subject hoping that one of them will match the quality standards of the photographer, namely in terms of illumination, focus and framing. Assuming that the framing issue is easily solved by cropping the photo, there is still the need to select which of the well framed photos, technically similar in terms of illumination and focus, are going to be kept (and in opposition which photos are going to be discarded). The process of visual observation, on a computer screen, in order to select the best photo is inaccurate and thus becomes a generator of insecurity feelings that may lead to no discarding of photos at all. In this work, we propose to address the issue of how to help the amateur photographer to select the best photo from a set of similar photos by analysing them in technical terms. The result is a novel workflow supported by a software package, guided by user input, which will allow the sorting of the similar photos accordingly to their technical characteristics (illumination and focus) and the user requirements. As a result, we expect that the process of choosing the best photo, and discarding of the remaining, becomes reliable and more comfortable

Soundtrack recommendation for images

The drastic increase in production of multimedia content has emphasized the research concerning its organization and retrieval. In this thesis, we address the problem of music retrieval when a set of images is given as input query, i.e., the problem of soundtrack recommendation for images. The task at hand is to recommend appropriate music to be played during the presentation of a given set of query images. To tackle this problem, we formulate a hypothesis that the knowledge appropriate for the task is contained in publicly available contemporary movies. Our approach, Picasso, employs similarity search techniques inside the image and music domains, harvesting movies to form a link between the domains. To achieve a fair and unbiased comparison between different soundtrack recommendation approaches, we proposed an evaluation benchmark. The evaluation results are reported for Picasso and the baseline approach, using the proposed benchmark. We further address two efficiency aspects that arise from the Picasso approach. First, we investigate the problem of processing top-K queries with set-defined selections and propose an index structure that aims at minimizing the query answering latency. Second, we address the problem of similarity search in high-dimensional spaces and propose two enhancements to the Locality Sensitive Hashing (LSH) scheme. We also investigate the prospects of a distributed similarity search algorithm based on LSH using the MapReduce framework. Finally, we give an overview of the PicasSound|a smartphone application based on the Picasso approach.Der drastische Anstieg von verfügbaren Multimedia-Inhalten hat die Bedeutung der Forschung über deren Organisation sowie Suche innerhalb der Daten hervorgehoben. In dieser Doktorarbeit betrachten wir das Problem der Suche nach geeigneten Musikstücken als Hintergrundmusik für Diashows. Wir formulieren die Hypothese, dass die für das Problem erforderlichen Kenntnisse in öffentlich zugänglichen, zeitgenössischen Filmen enthalten sind. Unser Ansatz, Picasso, verwendet Techniken aus dem Bereich der Ähnlichkeitssuche innerhalb von Bild- und Musik-Domains, um basierend auf Filmszenen eine Verbindung zwischen beliebigen Bildern und Musikstücken zu lernen. Um einen fairen und unvoreingenommenen Vergleich zwischen verschiedenen Ansätzen zur Musikempfehlung zu erreichen, schlagen wir einen Bewertungs-Benchmark vor. Die Ergebnisse der Auswertung werden, anhand des vorgeschlagenen Benchmarks, für Picasso und einen weiteren, auf Emotionen basierenden Ansatz, vorgestellt. Zusätzlich behandeln wir zwei Effizienzaspekte, die sich aus dem Picasso Ansatz ergeben. (i) Wir untersuchen das Problem der Ausführung von top-K Anfragen, bei denen die Ergebnismenge ad-hoc auf eine kleine Teilmenge des gesamten Indexes eingeschränkt wird. (ii) Wir behandeln das Problem der Ähnlichkeitssuche in hochdimensionalen Räumen und schlagen zwei Erweiterungen des Lokalitätssensitiven Hashing (LSH) Schemas vor. Zusätzlich untersuchen wir die Erfolgsaussichten eines verteilten Algorithmus für die Ähnlichkeitssuche, der auf LSH unter Verwendung des MapReduce Frameworks basiert. Neben den vorgenannten wissenschaftlichen Ergebnissen beschreiben wir ferner das Design und die Implementierung von PicassSound, einer auf Picasso basierenden Smartphone-Anwendung

Tiling Slideshow

這篇論文提出一種新的瀏覽媒介，複合式拼貼幻燈秀，以拼貼的方式展示照片，並配合襯底音樂的節拍播放。與傳統的照片幻燈秀比較，多張具相似特性的照片妥善地安排於相同版面中並播放。由科技寫作激發的靈感，每個版面配置由一張主題照片與多張修飾照片所組成。基於這想法，本幻燈秀系統由三個主要部分構成︰影像群集，音樂分析，版面配置。受限於展示的空間有限，我們考慮照片間的內容以及彼此的關係，並且將版面組織的轉變為一個條件最佳化的問題。 與單張照片播放的幻燈秀比較，實驗結果顯示我們的方法更能帶給使用者愉悅的照片瀏覽經驗。This thesis presents a new medium, called tiling slideshow, to display photos in a tile-like manner, coordinating with the pace of background music. In contrast to conventional photo slideshow, multiple photos that have similar characteristics are well arranged and displayed at the same layout. Motivated by the guidelines of technical writing, each displaying layout is composed of a larger topic photo and several small-size supportive photos. Based on this idea, the proposed tiling slideshow system consists of three major components: image clustering, music analyzer, and layout organizer. Given the limited displaying space, we consider the context and relationship between photos and model the layout organization as a constrainted optimization problem. Experiments on real consumer photograph collections show that the novel displaying method gives users more pleasant browsing experience than the methods that focus only on single photograph display.1 Introduction ........................................... 1 1.1 Motivation ........................................... 1 1.2 Related Works ........................................ 3 1.3 The Proposed Solution ................................ 4 1.4 Thesis Organization .................................. 6 2 System Overview ........................................ 7 2.1 Essential Idea ....................................... 7 2.2 System Framework ..................................... 8 3 Visual Processing ...................................... 13 3.1 Photo Preprocess ..................................... 13 3.1.1 Orientation Correction ............................. 13 3.1.2 Underexposure/Overexposure Photo Detection ......... 15 3.1.3 Duplicate Photo Detection .......................... 15 3.1.4 Blur Photo Detection ............................... 16 3.2 Image Clustering ..................................... 19 3.2.1 Time-based Clustering .............................. 20 3.2.2 Content-based Clustering ........................... 22 3.3 Region of Interest Determination ..................... 27 3.3.1 Region of Interest ................................. 27 3.3.2 Bottom-up Attention Detection ...................... 28 3.3.3 Top-down Attention Detection ....................... 31 3.4 Summary .............................................. 32 4 Music Analysis ......................................... 33 4.1 Beat Detection ....................................... 33 4.2 Music Segmentation ................................... 37 4.3 Summary .............................................. 39 5 Tiling Slideshow Composition ........................... 40 5.1 Photo Importance ..................................... 42 5.1.1 Cluster-based Importance ........................... 42 5.1.2 Photo-based Importance ............................. 43 5.2 Cluster Selection .................................... 44 5.3 Tiling Frame Generation .............................. 44 5.4 Template Importance .................................. 45 5.5 Template Determination ............................... 46 5.6 Composition .......................................... 47 5.6.1 Region Selection ................................... 48 5.6.2 Implementation ..................................... 49 5.6.3 Discussion ......................................... 50 6 Experimental Results ................................... 51 6.1 The Photo Content Set ............................... 51 6.2 The Subjective User Evaluation ....................... 52 7 Conclusions and Future Work ............................ 56 7.1 Conclusions .......................................... 56 7.2 Future Work .......................................... 57 References ............................................... 5