Utilization of multimodal interaction signals for automatic summarisation of academic presentations

Multimedia archives are expanding rapidly. For these, there exists a shortage of retrieval and summarisation techniques for accessing and browsing content where the main information exists in the audio stream. This thesis describes an investigation into the development of novel feature extraction and summarisation techniques for audio-visual recordings of academic presentations. We report on the development of a multimodal dataset of academic presentations. This dataset is labelled by human annotators to the concepts of presentation ratings, audience engagement levels, speaker emphasis, and audience comprehension. We investigate the automatic classification of speaker ratings and audience engagement by extracting audio-visual features from video of the presenter and audience and training classifiers to predict speaker ratings and engagement levels. Following this, we investigate automatic identi�cation of areas of emphasised speech. By analysing all human annotated areas of emphasised speech, minimum speech pitch and gesticulation are identified as indicating emphasised speech when occurring together. Investigations are conducted into the speaker's potential to be comprehended by the audience. Following crowdsourced annotation of comprehension levels during academic presentations, a set of audio-visual features considered most likely to affect comprehension levels are extracted. Classifiers are trained on these features and comprehension levels could be predicted over a 7-class scale to an accuracy of 49%, and over a binary distribution to an accuracy of 85%. Presentation summaries are built by segmenting speech transcripts into phrases, and using keywords extracted from the transcripts in conjunction with extracted paralinguistic features. Highest ranking segments are then extracted to build presentation summaries. Summaries are evaluated by performing eye-tracking experiments as participants watch presentation videos. Participants were found to be consistently more engaged for presentation summaries than for full presentations. Summaries were also found to contain a higher concentration of new information than full presentations

From open data to data-intensive science through CERIF

OGD (Open Government Data) is provided from government departments for transparency and to stimulate a market in ICT services for industry and citizens. Research datasets from publicly funded research commonly are associated with the open scholarly publications movement. However, the former world commonly is derived from the latter with generalisation and summarisation. There is advantage in a user of OGD being able to ‘drill down’ to the underlying research datasets. OGD encourages cross-domain research because the summarized data from different domains is more easily relatable. Bridging across the two worlds requires rich metadata; CERIF (Common European research Information Format) has proved itself to be ideally suited to this requirement. Utilising the research datasets is data-intensive science, a component of e-Research. Data-intensive science also requires access to an e-infrastructure. Virtualisation of this e-infrastructure optimizes this

From open data to data-intensive science through CERIF

OGD (Open Government Data) is provided from government departments for transparency and to stimulate a market in ICT services for industry and citizens. Research datasets from publicly funded research commonly are associated with the open scholarly publications movement. However, the former world commonly is derived from the latter with generalisation and summarisation. There is advantage in a user of OGD being able to ‘drill down’ to the underlying research datasets. OGD encourages cross-domain research because the summarized data from different domains is more easily relatable. Bridging across the two worlds requires rich metadata; CERIF (Common European research Information Format) has proved itself to be ideally suited to this requirement. Utilising the research datasets is data-intensive science, a component of e-Research. Data-intensive science also requires access to an e-infrastructure. Virtualisation of this e-infrastructure optimizes this

Collaborative learning with affective artificial study companions in a virtual learning environment

This research has been carried out in conjunction with Chapeltown and Harehills Assisted Learning Computer School (CHALCS) and local schools. CHALCS is an 'out-of-hours' school in a deprived inner-city community where unemployment is high and many children are failing to meet their educational potential. As the name implies CHALCS provides students with access to computers to support their learning. CHALCS relies on many volunteer tutors and specialist tutors are in short supply. This is especially true for subjects such as Advanced Level Physics with low numbers of students. This research aimed to investigate the feasibility of providing online study skills support to pupils at CHALCS and a local school. Research suggests that collaborative learning that prompts students to explain and justify their understanding can encourage deeper learning. As a potentially effective way of motivating deeper learning from hypertext course notes in a Virtual Learning Environment (VLE), this research investigates the feasibility of designing an artificial Agent capable of collaborating with the learner to jointly construct summary notes. Hypertext course notes covering a portion of the Advanced Level Physics curriculum were designed and uploaded into a WebCT based VLE. A specialist tutor validated the content of the course notes before the ease of use of the VLE was tested with target students. A study was then conducted to develop a model of the kinds of help students required in writing summary notes from the course-notes. Based on the derived process model of summarisation and an analysis of the content structure of the course notes, strategies for summarising the text were devised. An Animated Pedagogical Agent was designed incorporating these strategies. Two versions of the agent with opposing 'Affectations' (giving the appearance of different characters) were evaluated with users. It was therefore possible to test which artificial 'character' students preferred. From the evaluation study some conclusions are made concerning the effect of the two opposite characterisations on student perceptions of the agent and the degree to which it was helpful as a learning companion. Some recommendations for future work are then made

Automatic movie analysis and summarisation

Automatic movie analysis is the task of employing Machine Learning methods to the field of screenplays, movie scripts, and motion pictures to facilitate or enable various tasks throughout the entirety of a movie’s life-cycle. From helping with making informed decisions about a new movie script with respect to aspects such as its originality, similarity to other movies, or even commercial viability, all the way to offering consumers new and interesting ways of viewing the final movie, many stages in the life-cycle of a movie stand to benefit from Machine Learning techniques that promise to reduce human effort, time, or both. Within this field of automatic movie analysis, this thesis addresses the task of summarising the content of screenplays, enabling users at any stage to gain a broad understanding of a movie from greatly reduced data. The contributions of this thesis are four-fold: (i)We introduce ScriptBase, a new large-scale data set of original movie scripts, annotated with additional meta-information such as genre and plot tags, cast information, and log- and tag-lines. To our knowledge, Script- Base is the largest data set of its kind, containing scripts and information for almost 1,000 Hollywood movies. (ii) We present a dynamic summarisation model for the screenplay domain, which allows for extraction of highly informative and important scenes from movie scripts. The extracted summaries allow for the content of the original script to stay largely intact and provide the user with its important parts, while greatly reducing the script-reading time. (iii) We extend our summarisation model to capture additional modalities beyond the screenplay text. The model is rendered multi-modal by introducing visual information obtained from the actual movie and by extracting scenes from the movie, allowing users to generate visual summaries of motion pictures. (iv) We devise a novel end-to-end neural network model for generating natural language screenplay overviews. This model enables the user to generate short descriptive and informative texts that capture certain aspects of a movie script, such as its genres, approximate content, or style, allowing them to gain a fast, high-level understanding of the screenplay. Multiple automatic and human evaluations were carried out to assess the performance of our models, demonstrating that they are well-suited for the tasks set out in this thesis, outperforming strong baselines. Furthermore, the ScriptBase data set has started to gain traction, and is currently used by a number of other researchers in the field to tackle various tasks relating to screenplays and their analysis

Engaging the student voices to improve referencing skills and practices in higher education: A South African case study

Acknowledging that plagiarism includes many different behaviours, this study focusses on one specific plagiarism challenge, namely, acts of incorrect referencing and citation (sometimes called technical plagiarism). Notwithstanding anti-plagiarism policies being in place with penalties for a breach of the rules, Referencing Guides shared with students, additional workshops and tutorial sessions to facilitate the students' understanding of the Institution's referencing rules and standards, higher education institutions continue to experience an ongoing increase in the number of (technical) plagiarism cases, including repeat offenders. While the importance of proper referencing skills and techniques is fundamental to sound scholarship, a further concern is the impact of the accruing penalties on student success, retention, and graduateness. This study actively engaged the students, requiring them to reflect on their prior experiences with various sources of information, as well as their understanding and appreciation of how, why, what, and when to reference the resources used in their studies. The study discovered that students often describe the Institution's approach to teaching the skills of referencing as “alienating” and “punitive” and antithetical to learning. Based on the research data, and comparing the outcomes from other similar research studies, the paper propose leading practices that will support higher education institutions better prepare students, especially students of the Alpha Generation, for improved understanding and application of institutional norms for academic referencing and citation practices

Challenges and Opportunities in using Analytics Combined with Visualisation Techniques for Finding Anomalies in Digital Communications

Digital communication has changed human life since the invention of the internet. The growth of E-mail, social websites and other interpersonal communication systems in turn have brought rapid development in especially the key technological area of data analytics. Using advanced forms of analytics helps the examination of data and better informs investigative sense-making and decision-making of all kinds. The legal process called Electronic discovery (E-discovery) is used for investigating various events in the digital communication world, for the purpose of producing/obtaining evidence (such as evidence in the form of emails used in the Enron fraud case). Investigating digital communications collected over a period of time, manually, is a strenuous process, time consuming, expensive and not very effective. More recently, within E-discovery there has been development of analytics known in the legal community as “Technology assisted review” (TAR). TAR is a technologydriven assistant in E-discovery for identifying relevance in the documents/data which saves time and improves efficiency in investigation. At the same time, the efficacy of visualisation tools currently available in the market is increasing, where such tools depend on a combination of simple keyword searches and more complex representations (e.g. network graphs). Also in E-discovery, early case assessment is a process of estimating risk (cost and time) to prosecute or defend a legal case based on an early review of potentially relevant electronically stored information (ESI). Legal firms largely determine the duration of the E-discovery process and charge companies based on the volume of information collected and reviewed after an automated search, where ESI may then be manually reviewed intensely to determine relevance and privilege. This results in significant costs for the company or in a number of cases settlement because a party cannot afford to continue with the lawsuit due to Ediscovery costs. This paper examines some of the opportunities and challenges in searching digital communication data for E-discovery and investigations, and will explore how analytics coupled with visualisation techniques may lend support and guidance in these efforts. Addressing these combined techniques may yet yield improved data collection, analysis and understanding of how analysts/lawyers can work together using visualisations. In particular, we attempt to address two challenges: (i) improving comparison of subsets of data, and (ii) identifying anomalies (including sensitivities) in email communication

Generating Paths through Cultural Heritage Collections

Cultural heritage collections usually organise sets of items into exhibitions or guided tours. These items are often accompanied by text that describes the theme and topic of the exhibition and provides background context and details of connections with other items. The PATHS project brings the idea of guided tours to digital library collections where a tool to create virtual paths are used to assist with navigation and provide guides on particular subjects and topics. In this paper we characterise and analyse paths of items created by users of our online system. The analysis highlights that most users spend time selecting items relevant to their chosen topic, but few users took time to add background information to the paths. In order to address this, we conducted preliminary investigations to test whether Wikipedia can be used to automatically add background text for sequences of items. In the future we would like to explore the automatic creation of full paths