A Computational Model Of The Intelligibility Of American Sign Language Video And Video Coding Applications

Real-time, two-way transmission of American Sign Language (ASL) video over cellular networks provides natural communication among members of the Deaf community. Bandwidth restrictions on cellular networks and limited computational power on cellular devices necessitate the use of advanced video coding techniques designed explicitly for ASL video. As a communication tool, compressed ASL video must be evaluated according to the intelligibility of the conversation, not according to conventional definitions of video quality. The intelligibility evaluation can either be performed using human subjects participating in perceptual experiments or using computational models suitable for ASL video. This dissertation addresses each of these issues in turn, presenting a computational model of the intelligibility of ASL video, which is demonstrated to be accurate with respect to true intelligibility ratings as provided by human subjects. The computational model affords the development of video compression techniques that are optimized for ASL video. Guided by linguistic principles and human perception of ASL, this dissertation presents a full-reference computational model of intelligibility for ASL (CIM-ASL) that is suitable for evaluating compressed ASL video. The CIM-ASL measures distortions only in regions relevant for ASL communication, using spatial and temporal pooling mechanisms that vary the contribution of distortions according to their relative impact on the intelligibility of the compressed video. The model is trained and evaluated using ground truth experimental data, collected in three separate perceptual studies. The CIM-ASL provides accurate estimates of subjective intelligibility and demonstrates statistically significant improvements over computational models traditionally used to estimate video quality. The CIM-ASL is incorporated into an H.264/AVC compliant video coding framework, creating a closed-loop encoding system optimized explicitly for ASL intelligibility. This intelligibility optimized coder achieves bitrate reductions between 10% and 42% without reducing intelligibility, when compared to a general purpose H.264/AVC encoder. The intelligibility optimized encoder is refined by introducing reduced complexity encoding modes, which yield a 16% improvement in encoding speed. The purpose of the intelligibility optimized encoder is to generate video that is suitable for real-time ASL communication. Ultimately, the preferences of ASL users determine the success of the intelligibility optimized coder. User preferences are explicitly evaluated in a perceptual experiment in which ASL users select between the intelligibility optimized coder and a general purpose video coder. The results of this experiment demonstrate that the preferences vary depending on the demographics of the participants and that a significant proportion of users prefer the intelligibility optimized coder

Video quality requirements for South African Sign Language communications over mobile phones.

Includes abstract.Includes bibliographical references.This project aims to find the minimum video resolution and frame rate that supports intelligible cell phone based video communications in South African Sign Language

Semi-synchronous video for deaf telephony with an adapted synchronous codec

Magister Scientiae - MScCommunication tools such as text-based instant messaging, voice and video relay services, real-time video chat and mobile SMS and MMS have successfully been used among Deaf people. Several years of field research with a local Deaf community revealed that disadvantaged South African Deaf people preferred to communicate with both Deaf and hearing peers in South African Sign Language as opposed to text. Synchronous video chat and video relay services provided such opportunities. Both types of services are commonly available in developed regions, but not in developing countries like South Africa. This thesis reports on a workaround approach to design and develop an asynchronous video communication tool that adapted synchronous video codecs to store-and-forward video delivery. This novel asynchronous video tool provided high quality South African Sign Language video chat at the expense of some additional latency. Synchronous video codec adaptation consisted of comparing codecs, and choosing one to optimise in order to minimise latency and preserve video quality. Traditional quality of service metrics only addressed real-time video quality and related services. There was no such standard for asynchronous video communication. Therefore, we also enhanced traditional objective video quality metrics with subjective assessment metrics conducted with the local Deaf community.South Afric

JPEG XR scalable coding for remote image browsing applications

The growing popularity of the Internet has opened the road to multimedia and interactivity, emphasizing the importance of visual communication. In this context, digital images have taken a lead role and have an increasing number of applications. Consider, for example, the spread that digital cameras and mobile devices such as mobile phones have become in recent years. Thus, it arises the need for a flexible system that can handle images from different sources and are able to adapt to a different view. The importance of this issue lies in the application scenario: today there are datastores with a large number of images saved in JPEG format and systems for rendering digital images are various and with very different characteristics with each other. The ISO/IEC committee has recently issued a new format, called JPEG-XR, created explicitly for the modern digital cameras. The new coding algorithm JPEG-XR, can overcome various limitations of the first JPEG algorithm and provides viable alternatives to the JPEG2000 algorithm. This research has primarily focused on issues concerning the scalability of the new format of digital images.Additional scalability levels are fundamental for image browsing applications, because enable the system to ensure a correct and efficient functioning even when there is a sharp increase in the number of resources and users.Scalability is mostly required when dealing with large image database on the Web in order to reduce the transferred data, especially when it comes to large images. The interactive browsing also requires the ability to access to arbitrary parts of the image. The starting point is the use of a client-server architecture, in which the server stores a database of JPEG XR images and analyzes requests from a client. Client and server communicate via HTTP and use an exchange protocol. In order to minimize the transferred information, the JPEG XR coded file format should make use of the frequency mode order and partitioning of images into optimized tiles. The main goal is transmitting only some subset of the available sub-band coefficients. This is necessary to allow access an interactive access to portion of images, that are downloaded and displayed, minimizing the amount of data transferred and maintaining an acceptable image quality.The proposed architecture has of course prompted a study of errors in transmission on unreliable channel, such as the wireless one, and the definition of possible optimizations/variants of the codec in order to overcome its own limitations. Image data compressed with JPEG XR when transmitted over error-prone channels is severely distorted. In fact, due to the adaptive coding strategies used by the codec, even a single bit error causes a mismatch in the alignment of the reading position from the bit-stream, leading to completely different images at the decoder side. An extension to the JPEG XR algorithm is proposed, consisting in an error recovery process enabling the decoder to realign itself to the right bit-stream position and to correctly decode the most part of the image. Several experiments have been performed using different encoder parameter and different error probabilities while image distortion is measured by PSNR objective metric. The simplicity of the proposed algorithm adds very little computational overhead and seems very promising as confirmed by objective image quality results in experimental tests

Usability and content verification of a mobile tool to help a deaf person with pharmaceutical instruction

>Magister Scientiae - MScThis thesis describes a multi-disciplinary collaboration towards iterative development of a mobile communication tool to support a Deaf person in understanding usage directions for medication dispensed at a pharmacy. We are improving usability and correctness of the user interface. The tool translates medicine instruction given in English text to South African Sign Language videos, which are relayed to a Deaf user on a mobile phone. Communication between pharmacists and Deaf patients were studied to extract relevant exchanges between the two users. We incorporated the common elements of these dialogues to represent content in a veri able manner to ensure that the mobile tool relays the correct information to the Deaf user. Instructions are made available for a Deaf patient in sign language videos on a mobile device. A pharmacy setup was created to conduct trials of the tool with groups of end users, in order to collect usability data with recorded participant observation, questionnaires and focus group discussions. Subsequently, pre-recorded sign language videos, stored on a phone's memory card, were tested for correctness. Lastly we discuss the results and implications of the study and provide a conclusion to our research

Quality aspects of Internet telephony

Internet telephony has had a tremendous impact on how people communicate. Many now maintain contact using some form of Internet telephony. Therefore the motivation for this work has been to address the quality aspects of real-world Internet telephony for both fixed and wireless telecommunication. The focus has been on the quality aspects of voice communication, since poor quality leads often to user dissatisfaction. The scope of the work has been broad in order to address the main factors within IP-based voice communication. The first four chapters of this dissertation constitute the background material. The first chapter outlines where Internet telephony is deployed today. It also motivates the topics and techniques used in this research. The second chapter provides the background on Internet telephony including signalling, speech coding and voice Internetworking. The third chapter focuses solely on quality measures for packetised voice systems and finally the fourth chapter is devoted to the history of voice research. The appendix of this dissertation constitutes the research contributions. It includes an examination of the access network, focusing on how calls are multiplexed in wired and wireless systems. Subsequently in the wireless case, we consider how to handover calls from 802.11 networks to the cellular infrastructure. We then consider the Internet backbone where most of our work is devoted to measurements specifically for Internet telephony. The applications of these measurements have been estimating telephony arrival processes, measuring call quality, and quantifying the trend in Internet telephony quality over several years. We also consider the end systems, since they are responsible for reconstructing a voice stream given loss and delay constraints. Finally we estimate voice quality using the ITU proposal PESQ and the packet loss process. The main contribution of this work is a systematic examination of Internet telephony. We describe several methods to enable adaptable solutions for maintaining consistent voice quality. We have also found that relatively small technical changes can lead to substantial user quality improvements. A second contribution of this work is a suite of software tools designed to ascertain voice quality in IP networks. Some of these tools are in use within commercial systems today

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences