Designing novel applications for multimedia

Current R&D in Multimedia technology is advancing in a fierce rate and will sure to become part of our important regular items in a 'conventional' technology inventory in near future. While the R&D nature of this technology means its accuracy, reliability and robustness are not sufficient enough to be used in real world yet, we want to envision NOW the near-future where this technology will have matured and used in real applications in order to explore and start shaping many possible new ways this novel technology could be utilised. In this talk, some of this effort in designing novel applications that incorporate Multimedia technology as their backend will be presented. Looking at possible future applications of the outcomes of the Multimedia research will be useful for us in re-visiting our own initial ideas and visions when such technologies were first conceived. In addition, with some representative examples of such novel applications the problems and issues will be raised in interaction design research for novel applications and how somewhat different Human-Computer Interaction approach is required in practice

Issues in designing novel applications for multimedia technologies

Emerging computational multimedia tools and techniques promise powerful ways to organise, search and browse our ever-increasing multimedia contents by automating annotation and indexing, augmenting meta-data, understanding media contents, linking related pieces of information amongst them, and providing intriguing visualisation and exploration front-ends. Identifying real-world scenarios and designing interactive applications that leverage these developing multimedia technology is certainly an important research topic in itself but poses a number of challenges. In this talk, I will discuss and highlight some of these challenges in designing these novel applications by reflecting on my own design practice with a number of design examples

Clustering of Musical Pieces through Complex Networks: an Assessment over Guitar Solos

Musical pieces can be modeled as complex networks. This fosters innovative ways to categorize music, paving the way towards novel applications in multimedia domains, such as music didactics, multimedia entertainment and digital music generation. Clustering these networks through their main metrics allows grouping similar musical tracks. To show the viability of the approach, we provide results on a dataset of guitar solos.Comment: to appear in IEEE Multimedia magazin

Information extraction from multimedia web documents: an open-source platform and testbed

The LivingKnowledge project aimed to enhance the current state of the art in search, retrieval and knowledge management on the web by advancing the use of sentiment and opinion analysis within multimedia applications. To achieve this aim, a diverse set of novel and complementary analysis techniques have been integrated into a single, but extensible software platform on which such applications can be built. The platform combines state-of-the-art techniques for extracting facts, opinions and sentiment from multimedia documents, and unlike earlier platforms, it exploits both visual and textual techniques to support multimedia information retrieval. Foreseeing the usefulness of this software in the wider community, the platform has been made generally available as an open-source project. This paper describes the platform design, gives an overview of the analysis algorithms integrated into the system and describes two applications that utilise the system for multimedia information retrieval

Exploiting a new level of DLP in multimedia applications

This paper proposes and evaluates MOM: a novel ISA paradigm targeted at multimedia applications. By fusing conventional vector ISA approaches together with more recent SIMD-like (Single Instruction Multiple Data) ISAs (such as MMX), we have developed a new matrix oriented ISA which efficiently deals with the small matrix structures typically found in multimedia applications. MOM exploits a level of DLP not reachable by neither conventional vector ISAs nor SIMD-like media ISA extensions. Our results show that MOM provides a factor of 1.3x to 4x performance improvement when compared with two different multimedia extensions (MMX and MDMX) on several kernels, which translates into up to a 50% of performance gain when measuring full applications (20% in average). Furthermore, the streaming nature of MOM provides additional advantages for executing multimedia applications, such as a very low fetch pressure or a high tolerance to memory latency, making MOM an ideal candidate for the embedded domain.Peer ReviewedPostprint (published version

Saving Energy in Mobile Devices for On-Demand Multimedia Streaming -- A Cross-Layer Approach

This paper proposes a novel energy-efficient multimedia delivery system called EStreamer. First, we study the relationship between buffer size at the client, burst-shaped TCP-based multimedia traffic, and energy consumption of wireless network interfaces in smartphones. Based on the study, we design and implement EStreamer for constant bit rate and rate-adaptive streaming. EStreamer can improve battery lifetime by 3x, 1.5x and 2x while streaming over Wi-Fi, 3G and 4G respectively.Comment: Accepted in ACM Transactions on Multimedia Computing, Communications and Applications (ACM TOMCCAP), November 201

Designing novel applications for emerging multimedia technology

Current R&D in media technologies such as Multimedia, Semantic Web and Sensor Web technologies are advancing in a fierce rate and will sure to become part of our important regular items in a 'conventional' technology inventory in near future. While the R&D nature of these technologies means their accuracy, reliability and robustness are not sufficient enough to be used in real world yet, we want to envision now the near-future where these technologies will have matured and used in real applications in order to explore and start shaping many possible new ways these novel technologies could be utilised. In this talk, some of this effort in designing novel applications that incorporate various media technologies as their backend will be presented. Examples include novel scenarios of LifeLogging application that incorporate automatic structuring of millions of photos passively captured from a SenseCam (wearable digital camera that automatically takes photos triggered by environmental sensors) and an interactive TV application incorporating a number of multimedia tools yet extremely simple and easy to use with a remote control in a lean-back position. The talk will conclude with remarks on how the design of novel applications that have no precedence or existing user base should require somewhat different approach from those suggested and practiced in conventional usability engineering methodology

Fuzzy memoization for floating-point multimedia applications

Instruction memoization is a promising technique to reduce the power consumption and increase the performance of future low-end/mobile multimedia systems. Power and performance efficiency can be improved by reusing instances of an already executed operation. Unfortunately, this technique may not always be worth the effort due to the power consumption and area impact of the tables required to leverage an adequate level of reuse. In this paper, we introduce and evaluate a novel way of understanding multimedia floating-point operations based on the fuzzy computation paradigm: performance and power consumption can be improved at the cost of small precision losses in computation. By exploiting this implicit characteristic of multimedia applications, we propose a new technique called tolerant memoization. This technique expands the capabilities of classic memoization by associating entries with similar inputs to the same output. We evaluate this new technique by measuring the effect of tolerant memoization for floating-point operations in a low-power multimedia processor and discuss the trade-offs between performance and quality of the media outputs. We report energy improvements of 12 percent for a set of key multimedia applications with small LUT of 6 Kbytes, compared to 3 percent obtained using previously proposed techniques.Peer ReviewedPostprint (published version

Living with the Semantic Gap: Experiences and remedies in the context of medical imaging

Semantic annotation of images is a key concern for the newly emerged applications of semantic multimedia. Machine processable descriptions of images make it possible to automate a variety of tasks from search and discovery to composition and collage of image data bases. However, the ever occurring problem of the semantic gap between the low level descriptors and the high level interpretation of an image poses new challenges and needs to be addressed before the full potential of semantic multimedia can be realised. We explore the possibilities and lessons learnt with applied semantic multimedia from our engagement with medical imaging where we deployed ontologies and a novel distributed architecture to provide semantic annotation, decision support and methods for tackling the semantic gap problem