Study on Situation-oriented Classification of Sightseeing Images Based on Visual and Metadata Features

This thesis proposes a method for classifying sightseeing images into different situations based on their visual and metadata features. The widespread use of digital cameras and smart phones has brought about a situation where tourists take lots of photos of memorable moments during their travels and upload these photos to web albums such as Flickr or Picasa. These sightseeing images then become useful resources for others who plan to visit the places shown in the images. As scenes of sightseeing spots vary from situation to situation, the impression one gets from viewing these images depends heavily on conditions such as the weather and season. If a web-based tourist service could provide tourists with different views of sightseeing spots in various situations, visitors would be able to plan their vacations by looking at the views they enjoy. That is, such a service would be useful for tourists to plan when and where to visit. To achieve this goal, a method that can classify various sightseeing images into various situations is required. Although image classification / annotation using visual and text features is becoming a major research topic in various fields, such as information retrieval and web intelligence, image classification methods focusing on various situations have not been studied yet. One of the contributions of this thesis is to consider various situations and organize them in terms of their characteristics. The situations treated in this thesis are classified into weather-related, time-related, and season-related ones. Weather-related situations include sunshiny and cloudy situations, and color features of sky regions are expected to be effective as a means of classifying them. On the other hand, time-related situations are characterized as certain times of the day such as sunrise/sunset, daytime, and night-time. Therefore, shooting date and time, i.e., metadata attached to the photos, are important features for such a classification. Different from weather-related and time-related situations, scenery change by season will depend on the characteristics of a sightseeing spot. It may happen that even though two sightseeing spots are geographically close, one maybe season-dependent and the other not. Therefore, sightseeing spots should also be classified into season-dependent and season-independent as a preprocessing for image classification. This thesis proposes different classification methods for each of these situation types. The thesis consists of six chapters. Chapter 1 describes the background and motivation. The vast amount of sightseeing images available in the web albums is an important resource for tourists. The purpose of this thesis is to establish an efficient image classification method targeting sightseeing images showing various situations, which will add extra value to existing web-based tourist services. The related topics of the thesis, i.e., image classification / annotation, have attracted a lot of research, and various features and integration methods have been studied. However, the major focus of these studies has been general-purpose processing; methods focusing on various situations have not been studied yet. This chapter defines and organizes the situations to be handled in the thesis and discusses the challenges of classifying sightseeing images into each situation. Chapter 2 describes the existing applications of tourism informatics. Image classification and annotation methods based on supervised and unsupervised learning with various features are also covered as related work. Chapter 3 describes content-based image classification targeting weather-related and time-related situations. Visual features for identifying each target situation are considered from viewpoints such as composition of the photos and typical colors in each situation. The images are classified in a hierarchical manner, in each stage of which efficient color features, region of interests (ROI), and cluster identification method are determined. Experimental results show that the proposed method can obtain clusters for each situation with high precision and recall. Chapter 4 focuses on time-related situations and extends the content-based image classification method proposed in Chapter 3 by introducing filtering based on tag information. By using timestamps attached to images, clusters for the situations obtained by the content-based approach are verified to increase the accuracy of the classification. The time windows are adjusted by considering the geolocation of sightseeing spots, and this adjustment is based on information obtained from the Web. Experimental results show that this method can improve precision while maintaining recall in most cases. Chapter 5 focuses on season-related situations and proposes a method for classifying sightseeing spots into season-dependent and season-independent ones as preprocessing for image classification. If image processing is required in order to extract features from photos, the network load for downloading photos and the cost of image processing become a serious problem. To solve this problem, the statistical features of sightseeing spots calculated using metadata are proposed. Image processing is only applied to the spots classified as season-dependent by machine learning with the statistical features. Experimental results show that this method can classify actual sightseeing spots with high precision and recall. Chapter 6 summarizes the conclusions presented in Chapter 3 to Chapter 5. This thesis proposes three kinds of image classification methods, each of which employs efficient visual and metadata features and integration methods for the target situations. The results of this thesis are meant to contribute to tourism and related applications, which are important issues in many cities including Tokyo. As the volume of images and metadata available on the Web is still increasing at a rapid rate, the contributions of the thesis may have numerous other applications.首都大学東京, 2013-09-30, 博士(工学), 甲第437号首都大学東