Shot boundary detection in video sequence using multifractal analysis

Moderan i efikasan menadžment video sadržaja podrazumeva mogućnost pretraživanja videa na osnovu automatski generisanih oznaka i informacija dobijenih analizom sadržaja frejmova. Vremenskom segmentacijom videa generiše se hijerarhijska struktura video sadržaja, koja predstavlja osnovu za automatsku anotaciju videa. Prvi korak vremenske segmentacije predstavlja određivanje granica kadrova kao najniže hijerarhijske logičke strukture videa. Kadar predstavlja jedno kontinuirano snimanje kamerom između uključivanja i isključivanja kamere. Grupe kadrova kreiraju više hijejarhijske logičke strukture videa kao sto su: scene, sekvence i programi. Pri detekciji granica kadrova razlikuju se dva osnovna tipa tranzicije izmedju kadrova - nagle i gradijentne promene kadrova. Naglim promenama kadrova nazivaju se tranzicije između kadrova koje traju samo dva frejma, dok kod gradijentnih tranzicija prelazak između dva kadra traje više frejmova. Tema ove disertacije je realizacija algoritma za efikasnu detekciju naglih promena kadrova u video sekvencama različitog sadržaja, dinamike i nivoa postprodukcije. U tu svrhu, sadržaj frejmova video sekvenci opisan je obeležjima za boju i teksturu. Korišćeni su histogram boje i vejvlet obeležje kao elementarna obeležja izdvojena iz frejmova, na lokalnom i globalnom nivou. Odabrana obeležja obezbeđuju sa jedne strane dovoljnu osetljivost na nagle promene vizuelnog sadržaja na frejmovima. Sa druge strane, odabrana obeležja pokazuju određeni nivo tolerancije na promene sadržaja niskog intenziteta, kao što su: kretanje objekata, kamere i promene osvetljenja. Na obeležja frejmova primenjene su dve različite diferencijalne metrike, korelacija i multifraktalna analiza. Korelacija, kao diferencijalna metrika, kombinovanim pristupom je omogućila formiranje specifičnih artefakata koje je jednostavno detektovati u strukturi signala. Artefakti zbog dinamike promena unutar kadrova mogu biti degradirani i pomereni u odnosu na poziciju stvarne nagle promene kadra. Za lokalizaciju nagle promene sadržaja u vremenu primenjena je multifraktalna analiza, tako što je iskorišćena sposobnost ove tehnike da dobro opiše lokalnu strukturu signala. U domenu grubih Hölder-ovih eksponenata multifraktalne analize, nagla promena signala se prikazuje kao lokalni maksimum. Kombinacijom ove dve metrike (ponovi metrike) detektuju se pozicije kandidata naglih promena kadrova. Klasifikacija kandidata izvršena je primenom klasifikatora baziranih na fiksnim pragovima i primenom dobro definisanih pravila. Kao dodatna mera u cilju smanjenja osetljivosti sistema na promene osvetljenja, u algoritam je implementiran i flesh detektor. Realizovani algoritam za detekciju naglih promena kadrova testiran je na velikom broju frejmova (>1.500.000 frejmova), video sekvenci različitih sadržaja, žanrova i tehnologija proizvodnje. Algoritam je pokazao visoke performanse, odnosno tačnost i preciznost, među kojima se posebno ističe tačnost. Pokazalo se da preciznost algoritma zavisi od klase video sekvenci nad kojom je obavljeno testiranje. U disertaciji je prikazano poređenje realizovanog algoritma sa algoritmima za detekciju naglih promena kadrova visokih performansi iz literature. Poređenja su vršena na osnovu tačnosti, preciznosti i F-mere. Rezultati poređenja su pokazali da je predloženi algoritam porediv po performansama sa ostalim algoritmima, a za neke video sekvence pokazao je bolje rezultate od ostalih. Prikazani rezultati ukazali su na velike mogućnosti predloženog algoritma u pogledu daljeg razvoja.Modern and efficient management of video content implies the possibility of video searching on the basis of automatically generated labels and information gained by the analysis of frame content. Temporal video segmentation generates hierarchical structure of video. This structure represents the basis for the automated annotation of video content. The first step of temporal segmentation is the determination of shot boundaries, as the lowest hierarchical logical structure of video. A shot refers to a continual recording with video camera between switch-on and switch-off of camera. Groups of shots create higher hierarchical logical structures of video, such as: scenes, sequences and programs. In the shot boundary detection there are two basic types of transitions between shots – abrupt and gradual shot changes. Abrupt shot changes refer to those transitions between shots which last only two frames, whereas in gradual shot changes the transitions between two shots last several frames. The topic of this thesis is the experimental realization of algorithm for efficient abrupt shot changes detection in video sequences of different contents, dynamics and postproduction levels. For this purpose, the frame content of video sequences is described with color and texture features. Color histogram and wavelet feature are used as basic features extracted from frames, on both local and global levels. On the one hand, the selected features provide sufficient sensibility regarding sudden changes of visual content on frames. On the other hand, the selected features show certain level of tolerance for the low intensity content changes, such as: moving of objects, camera and light changes. Correlation and multifractal analysis have been used on frame features as differential metrics. With combined approach, correlation as a differential metrics applied on frames features, has enabled the formation of specific artefacts which can be easily detected in signal structure. Due to the change dynamics within shots, artefacts can be degraded and shifted regarding the position of real abrupt shot changes. Multifractal analysis has been used for temporal localization of abrupt content changes due to its ability to provide a detailed description of signal local structure. In the domain of rough Hölder multifractal analysis exponents, abrupt signal change is shown as a local maximum. By combining these two metrics, correlation and multifractal analysis, the positions of abrupt shot change candidates are detected. Candidate classification has been carried out by means of classifiers based on fixed thresholds and predefined rules. As an additional precaution aimed at reducing system sensibility related to light changes, flash detector has been implemented into the algorithm. The implemented algorithm for the abrupt shot change detection has been tested on a large number of frames (>1.500.000 frames), video sequences of different contents, genres and production technologies. The algorithm has shown high performances, i.e. recall and precision, especially recall. It has been shown that precision of algorithm depends on the class of video sequence being tested. In this thesis, the comparisons between the proposed algorithm and the high-performance algorithms for the detection of abrupt shot changes in literature have been presented. Comparisons have been carried out on the basis of recall, precision and F-measure. The findings have shown that proposed algorithm is comparable by performances with other algorithms, showing even better results for some video sequences. The achieved results indicate great possibilities of the proposed algorithm for further development

Shot boundary detection in video sequence using multifractal analysis

Moderan i efikasan menadžment video sadržaja podrazumeva mogućnost pretraživanja videa na osnovu automatski generisanih oznaka i informacija dobijenih analizom sadržaja frejmova. Vremenskom segmentacijom videa generiše se hijerarhijska struktura video sadržaja, koja predstavlja osnovu za automatsku anotaciju videa. Prvi korak vremenske segmentacije predstavlja određivanje granica kadrova kao najniže hijerarhijske logičke strukture videa. Kadar predstavlja jedno kontinuirano snimanje kamerom između uključivanja i isključivanja kamere. Grupe kadrova kreiraju više hijejarhijske logičke strukture videa kao sto su: scene, sekvence i programi. Pri detekciji granica kadrova razlikuju se dva osnovna tipa tranzicije izmedju kadrova - nagle i gradijentne promene kadrova. Naglim promenama kadrova nazivaju se tranzicije između kadrova koje traju samo dva frejma, dok kod gradijentnih tranzicija prelazak između dva kadra traje više frejmova. Tema ove disertacije je realizacija algoritma za efikasnu detekciju naglih promena kadrova u video sekvencama različitog sadržaja, dinamike i nivoa postprodukcije. U tu svrhu, sadržaj frejmova video sekvenci opisan je obeležjima za boju i teksturu. Korišćeni su histogram boje i vejvlet obeležje kao elementarna obeležja izdvojena iz frejmova, na lokalnom i globalnom nivou. Odabrana obeležja obezbeđuju sa jedne strane dovoljnu osetljivost na nagle promene vizuelnog sadržaja na frejmovima. Sa druge strane, odabrana obeležja pokazuju određeni nivo tolerancije na promene sadržaja niskog intenziteta, kao što su: kretanje objekata, kamere i promene osvetljenja. Na obeležja frejmova primenjene su dve različite diferencijalne metrike, korelacija i multifraktalna analiza. Korelacija, kao diferencijalna metrika, kombinovanim pristupom je omogućila formiranje specifičnih artefakata koje je jednostavno detektovati u strukturi signala. Artefakti zbog dinamike promena unutar kadrova mogu biti degradirani i pomereni u odnosu na poziciju stvarne nagle promene kadra. Za lokalizaciju nagle promene sadržaja u vremenu primenjena je multifraktalna analiza, tako što je iskorišćena sposobnost ove tehnike da dobro opiše lokalnu strukturu signala. U domenu grubih Hölder-ovih eksponenata multifraktalne analize, nagla promena signala se prikazuje kao lokalni maksimum. Kombinacijom ove dve metrike (ponovi metrike) detektuju se pozicije kandidata naglih promena kadrova. Klasifikacija kandidata izvršena je primenom klasifikatora baziranih na fiksnim pragovima i primenom dobro definisanih pravila. Kao dodatna mera u cilju smanjenja osetljivosti sistema na promene osvetljenja, u algoritam je implementiran i flesh detektor. Realizovani algoritam za detekciju naglih promena kadrova testiran je na velikom broju frejmova (>1.500.000 frejmova), video sekvenci različitih sadržaja, žanrova i tehnologija proizvodnje. Algoritam je pokazao visoke performanse, odnosno tačnost i preciznost, među kojima se posebno ističe tačnost. Pokazalo se da preciznost algoritma zavisi od klase video sekvenci nad kojom je obavljeno testiranje. U disertaciji je prikazano poređenje realizovanog algoritma sa algoritmima za detekciju naglih promena kadrova visokih performansi iz literature. Poređenja su vršena na osnovu tačnosti, preciznosti i F-mere. Rezultati poređenja su pokazali da je predloženi algoritam porediv po performansama sa ostalim algoritmima, a za neke video sekvence pokazao je bolje rezultate od ostalih. Prikazani rezultati ukazali su na velike mogućnosti predloženog algoritma u pogledu daljeg razvoja.Modern and efficient management of video content implies the possibility of video searching on the basis of automatically generated labels and information gained by the analysis of frame content. Temporal video segmentation generates hierarchical structure of video. This structure represents the basis for the automated annotation of video content. The first step of temporal segmentation is the determination of shot boundaries, as the lowest hierarchical logical structure of video. A shot refers to a continual recording with video camera between switch-on and switch-off of camera. Groups of shots create higher hierarchical logical structures of video, such as: scenes, sequences and programs. In the shot boundary detection there are two basic types of transitions between shots – abrupt and gradual shot changes. Abrupt shot changes refer to those transitions between shots which last only two frames, whereas in gradual shot changes the transitions between two shots last several frames. The topic of this thesis is the experimental realization of algorithm for efficient abrupt shot changes detection in video sequences of different contents, dynamics and postproduction levels. For this purpose, the frame content of video sequences is described with color and texture features. Color histogram and wavelet feature are used as basic features extracted from frames, on both local and global levels. On the one hand, the selected features provide sufficient sensibility regarding sudden changes of visual content on frames. On the other hand, the selected features show certain level of tolerance for the low intensity content changes, such as: moving of objects, camera and light changes. Correlation and multifractal analysis have been used on frame features as differential metrics. With combined approach, correlation as a differential metrics applied on frames features, has enabled the formation of specific artefacts which can be easily detected in signal structure. Due to the change dynamics within shots, artefacts can be degraded and shifted regarding the position of real abrupt shot changes. Multifractal analysis has been used for temporal localization of abrupt content changes due to its ability to provide a detailed description of signal local structure. In the domain of rough Hölder multifractal analysis exponents, abrupt signal change is shown as a local maximum. By combining these two metrics, correlation and multifractal analysis, the positions of abrupt shot change candidates are detected. Candidate classification has been carried out by means of classifiers based on fixed thresholds and predefined rules. As an additional precaution aimed at reducing system sensibility related to light changes, flash detector has been implemented into the algorithm. The implemented algorithm for the abrupt shot change detection has been tested on a large number of frames (>1.500.000 frames), video sequences of different contents, genres and production technologies. The algorithm has shown high performances, i.e. recall and precision, especially recall. It has been shown that precision of algorithm depends on the class of video sequence being tested. In this thesis, the comparisons between the proposed algorithm and the high-performance algorithms for the detection of abrupt shot changes in literature have been presented. Comparisons have been carried out on the basis of recall, precision and F-measure. The findings have shown that proposed algorithm is comparable by performances with other algorithms, showing even better results for some video sequences. The achieved results indicate great possibilities of the proposed algorithm for further development

Shot boundary detection in video sequence using multifractal analysis

Moderan i efikasan menadžment video sadržaja podrazumeva mogućnost pretraživanja videa na osnovu automatski generisanih oznaka i informacija dobijenih analizom sadržaja frejmova. Vremenskom segmentacijom videa generiše se hijerarhijska struktura video sadržaja, koja predstavlja osnovu za automatsku anotaciju videa. Prvi korak vremenske segmentacije predstavlja određivanje granica kadrova kao najniže hijerarhijske logičke strukture videa. Kadar predstavlja jedno kontinuirano snimanje kamerom između uključivanja i isključivanja kamere. Grupe kadrova kreiraju više hijejarhijske logičke strukture videa kao sto su: scene, sekvence i programi. Pri detekciji granica kadrova razlikuju se dva osnovna tipa tranzicije izmedju kadrova - nagle i gradijentne promene kadrova. Naglim promenama kadrova nazivaju se tranzicije između kadrova koje traju samo dva frejma, dok kod gradijentnih tranzicija prelazak između dva kadra traje više frejmova. Tema ove disertacije je realizacija algoritma za efikasnu detekciju naglih promena kadrova u video sekvencama različitog sadržaja, dinamike i nivoa postprodukcije. U tu svrhu, sadržaj frejmova video sekvenci opisan je obeležjima za boju i teksturu. Korišćeni su histogram boje i vejvlet obeležje kao elementarna obeležja izdvojena iz frejmova, na lokalnom i globalnom nivou. Odabrana obeležja obezbeđuju sa jedne strane dovoljnu osetljivost na nagle promene vizuelnog sadržaja na frejmovima. Sa druge strane, odabrana obeležja pokazuju određeni nivo tolerancije na promene sadržaja niskog intenziteta, kao što su: kretanje objekata, kamere i promene osvetljenja. Na obeležja frejmova primenjene su dve različite diferencijalne metrike, korelacija i multifraktalna analiza. Korelacija, kao diferencijalna metrika, kombinovanim pristupom je omogućila formiranje specifičnih artefakata koje je jednostavno detektovati u strukturi signala. Artefakti zbog dinamike promena unutar kadrova mogu biti degradirani i pomereni u odnosu na poziciju stvarne nagle promene kadra. Za lokalizaciju nagle promene sadržaja u vremenu primenjena je multifraktalna analiza, tako što je iskorišćena sposobnost ove tehnike da dobro opiše lokalnu strukturu signala. U domenu grubih Hölder-ovih eksponenata multifraktalne analize, nagla promena signala se prikazuje kao lokalni maksimum. Kombinacijom ove dve metrike (ponovi metrike) detektuju se pozicije kandidata naglih promena kadrova. Klasifikacija kandidata izvršena je primenom klasifikatora baziranih na fiksnim pragovima i primenom dobro definisanih pravila. Kao dodatna mera u cilju smanjenja osetljivosti sistema na promene osvetljenja, u algoritam je implementiran i flesh detektor. Realizovani algoritam za detekciju naglih promena kadrova testiran je na velikom broju frejmova (>1.500.000 frejmova), video sekvenci različitih sadržaja, žanrova i tehnologija proizvodnje. Algoritam je pokazao visoke performanse, odnosno tačnost i preciznost, među kojima se posebno ističe tačnost. Pokazalo se da preciznost algoritma zavisi od klase video sekvenci nad kojom je obavljeno testiranje. U disertaciji je prikazano poređenje realizovanog algoritma sa algoritmima za detekciju naglih promena kadrova visokih performansi iz literature. Poređenja su vršena na osnovu tačnosti, preciznosti i F-mere. Rezultati poređenja su pokazali da je predloženi algoritam porediv po performansama sa ostalim algoritmima, a za neke video sekvence pokazao je bolje rezultate od ostalih. Prikazani rezultati ukazali su na velike mogućnosti predloženog algoritma u pogledu daljeg razvoja.Modern and efficient management of video content implies the possibility of video searching on the basis of automatically generated labels and information gained by the analysis of frame content. Temporal video segmentation generates hierarchical structure of video. This structure represents the basis for the automated annotation of video content. The first step of temporal segmentation is the determination of shot boundaries, as the lowest hierarchical logical structure of video. A shot refers to a continual recording with video camera between switch-on and switch-off of camera. Groups of shots create higher hierarchical logical structures of video, such as: scenes, sequences and programs. In the shot boundary detection there are two basic types of transitions between shots – abrupt and gradual shot changes. Abrupt shot changes refer to those transitions between shots which last only two frames, whereas in gradual shot changes the transitions between two shots last several frames. The topic of this thesis is the experimental realization of algorithm for efficient abrupt shot changes detection in video sequences of different contents, dynamics and postproduction levels. For this purpose, the frame content of video sequences is described with color and texture features. Color histogram and wavelet feature are used as basic features extracted from frames, on both local and global levels. On the one hand, the selected features provide sufficient sensibility regarding sudden changes of visual content on frames. On the other hand, the selected features show certain level of tolerance for the low intensity content changes, such as: moving of objects, camera and light changes. Correlation and multifractal analysis have been used on frame features as differential metrics. With combined approach, correlation as a differential metrics applied on frames features, has enabled the formation of specific artefacts which can be easily detected in signal structure. Due to the change dynamics within shots, artefacts can be degraded and shifted regarding the position of real abrupt shot changes. Multifractal analysis has been used for temporal localization of abrupt content changes due to its ability to provide a detailed description of signal local structure. In the domain of rough Hölder multifractal analysis exponents, abrupt signal change is shown as a local maximum. By combining these two metrics, correlation and multifractal analysis, the positions of abrupt shot change candidates are detected. Candidate classification has been carried out by means of classifiers based on fixed thresholds and predefined rules. As an additional precaution aimed at reducing system sensibility related to light changes, flash detector has been implemented into the algorithm. The implemented algorithm for the abrupt shot change detection has been tested on a large number of frames (>1.500.000 frames), video sequences of different contents, genres and production technologies. The algorithm has shown high performances, i.e. recall and precision, especially recall. It has been shown that precision of algorithm depends on the class of video sequence being tested. In this thesis, the comparisons between the proposed algorithm and the high-performance algorithms for the detection of abrupt shot changes in literature have been presented. Comparisons have been carried out on the basis of recall, precision and F-measure. The findings have shown that proposed algorithm is comparable by performances with other algorithms, showing even better results for some video sequences. The achieved results indicate great possibilities of the proposed algorithm for further development

Classification of Prolapsed Mitral Valve versus Healthy Heart from Phonocardiograms by Multifractal Analysis

Phonocardiography has shown a great potential for developing low-cost computer-aided diagnosis systems for cardiovascular monitoring. So far, most of the work reported regarding cardiosignal analysis using multifractals is oriented towards heartbeat dynamics. This paper represents a step towards automatic detection of one of the most common pathological syndromes, so-called mitral valve prolapse (MVP), using phonocardiograms and multifractal analysis. Subtle features characteristic for MVP in phonocardiograms may be difficult to detect. The approach for revealing such features should be locally based rather than globally based. Nevertheless, if their appearances are specific and frequent, they can affect a multifractal spectrum. This has been the case in our experiment with the click syndrome. Totally, 117 pediatric phonocardiographic recordings (PCGs), 8 seconds long each, obtained from 117 patients were used for PMV automatic detection. We propose a two-step algorithm to distinguish PCGs that belong to children with healthy hearts and children with prolapsed mitral valves (PMVs). Obtained results show high accuracy of the method. We achieved 96.91% accuracy on the dataset (97 recordings). Additionally, 90% accuracy is achieved for the evaluation dataset (20 recordings). Content of the datasets is confirmed by the echocardiographic screening

Identification of S1 and S2 Heart Sound Patterns Based on Fractal Theory and Shape Context

There has been a sustained effort in the research community over the recent years to develop algorithms that automatically analyze heart sounds. One of the major challenges is identifying primary heart sounds, S1 and S2, as they represent reference events for the analysis. The study presented in this paper analyzes the possibility of improving the structure characterization based on shape context and structure assessment using a small number of descriptors. Particularly, for the primary sound characterization, an adaptive waveform filtering is applied based on blanket fractal dimension for each preprocessed sound candidate belonging to pediatric subjects. This is followed by applying the shape based methods selected for the structure assessment of primary heart sounds. Different methods, such as the fractal ones, are used for the comparison. The analysis of heart sound patterns is performed using support vector machine classifier showing promising results (above 95% accuracy). The obtained results suggest that it is possible to improve the identification process using the shape related methods which are rarely applied. This can be helpful for applications involving automatic heart sound analysis

Adaptive content-based image retrieval with relevance feedback

Abstract — Retrieval of images, based on similarities between feature vectors of querying image and those from database, is considered. The searching procedure was performed through the two basic steps: an objective one, based on the Euclidean distances and a subjective one based on the user’s relevance feedback. Images recognized from user as the best matched to a query are labeled and used for updating the query feature vector through a RBF (radial basis function) neural network. The searching process is repeated from such subjectively refined feature vectors. In practice, several iterative steps are sufficient, as confirmed by intensive simulations

Systems with Relevance Feedback

Abstract — A content-based image retrieval system where an active learning strategy is used to gain relevance feedback (RF) is described. In this way retrieving process may be highly accelerated without significant degradation of accuracy Searching procedure was performed through the two basic steps: an objective one, based on the Euclidean distances and a subjective one based on the user’s relevance feedback. Images recognized from user as the best matched to a query are labeled and used for updating the query feature vector through a RBF (radial basis function) neural network. In this process user change feature vector which became more refined and appropriate for future search. In practice, several iterative steps are sufficient, as confirmed by intensive simulations. Index Terms — Content-based image retrieval, low level image descriptors, neural network decision, relevance feedback, subjective perception of images