Real-time hallucination simulation and sonification through user-led development of an iPad augmented reality performance

The simulation of visual hallucinations has multiple applications. The authors present a new approach to hallucination simulation, initially developed for a performance, that proved to have uses for individuals suffering from certain types of hallucinations. The system, originally developed with a focus on the visual symptoms of palinopsia experienced by the lead author, allows real-time visual expression using augmented reality via an iPad. It also allows the hallucinations to be converted into sound through visuals sonification. Although no formal experimentation was conducted, the authors report on a number of unsolicited informal responses to the simulator from palinopsia sufferers and the Palinopsia Foundation

The Glitch Aesthetic

The miscommunication between sender and receiver during transcoding indexes specific historical moments similarly to analog film\u27s indexical trace. Iconography and glitch art begin to establish glitch\u27s deictic index. The glitch aesthetic exposes societal paranoia by illustrating dependence on the digital and fear of system failure. With the advent of video sharing sites like Youtube and popular cyberfilms, the glitch aesthetic has evolved into a pop culture artifact

Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model

Current captioning approaches tend to generate correct but "generic" descriptions that lack real-world knowledge, e.g., named entities and contextual information. Considering that Vision-Language Pre-Training (VLP) models master massive such knowledge from large-scale web-harvested data, it is promising to utilize the generalizability of VLP models to incorporate knowledge into image descriptions. However, using VLP models faces challenges: zero-shot inference suffers from knowledge hallucination that leads to low-quality descriptions, but the generic bias in downstream task fine-tuning hinders the VLP model from expressing knowledge. To address these concerns, we propose a simple yet effective method called Knowledge-guided Replay (K-Replay), which enables the retention of pre-training knowledge during fine-tuning. Our approach consists of two parts: (1) a knowledge prediction task on automatically collected replay exemplars to continuously awaken the VLP model's memory about knowledge, thus preventing the model from collapsing into the generic pattern; (2) a knowledge distillation constraint to improve the faithfulness of generated descriptions hence alleviating the knowledge hallucination. To evaluate knowledge-enhanced descriptions, we construct a novel captioning benchmark KnowCap, containing knowledge of landmarks, famous brands, special foods and movie characters. Experimental results show that our approach effectively incorporates knowledge into descriptions, outperforming strong VLP baseline by 20.9 points (78.7->99.6) in CIDEr score and 20.5 percentage points (34.0%->54.5%) in knowledge recognition accuracy. Our code and data is available at https://github.com/njucckevin/KnowCap.Comment: Accepted at ACM Multimedia (ACMMM) 202

Taking a Joke Seriously: Mickey Mouse and William Kentridge

Tracing the similarities in technical support shared by South African visual artist William Kentridge’s Nine Drawings for Projection and early cel animation leads me back to Walter Benjamin’s enigmatic invocation of Mickey Mouse in the second version of “The Work of Art in the Age of Its Technological Reproducibility.” If Benjamin here ascribes the role of a collective dream-figure to Mickey Mouse, thereby conflating the Freudian concept of the asocial dream with the wholly social joke, I argue that in his work, Kentridge wants to take the joke seriously – that is, work within and experiment with the rules, forms of perception, and universal language trained by the medium of early Disney animation

Digital unreason: Jordan Wolfson’s ‘Evil Jew’

This article privileges an analysis of the artwork Animation, masks (2011) by the Jewish-American contemporary artist Jordan Wolfson. This animated film is over 10 minutes long and features only one character, based on a collection of references Wolfson found under the Google search terms ‘evil Jew’ and ‘Shylock’. One of the key references for this ‘evil Jew’ stereotype, ‘Le Happy Merchant’, is a popular meme circulating on the 4chan media network and has become a mascot of the so called ‘alt-right’ – a loose network of racists and anti-Semites that became militant supporters of the 2016 Donald Trump presidential campaign and presidency. In Wolfson’s version of the meme, professional American animators were hired to give this stereotype the illusion of life. The result is a fascist stereotype restaged as an endearing Dreamworks character, unpredictable and incoherent, lurching between violent and placatory gestures. Rather than attempt a close analysis, this article argues that Wolfson’s social link to stereotype, conspiracy theory and fascist rebellion requires greater interrogation in light of the Donald Trump campaign, Brexit and the Le Pen candidacy. In all these campaigns, a vitalistic critique of metropolitan ‘elites’ or ‘globalists’ became the anti-establishment pretext for verbal assaults on immigrants, people of colour and Muslims. Alberto Toscano and Jeff Kinkle in Cartographies of the Absolute (2015) connect a popular upsurge in conspiratorial thinking to an absented critique of capitalism. They claim that if the abstract and impersonal character of capitalism is never awarded popular interrogation, conspiracy theory is an easy substitute, leaving the door open to anti-Semitic and racist projections. In turn, they call for art and popular culture that creates a systematic link to the bewildering movements of late capitalism, rather than fall into conspiratorial thinking. In this article, the author suggests that Wolfson in Animation, masks is using animation and caricature to bear witness to the circulation of anti-Semitic conspiracy and the narcissistic social networks that have been emboldened by the rise of the international far Right. However, by ambivalently playing around with stereotype, Wolfson also risks falling into complicity with forces of unreason that are indifferent to ironic treatment. This article discusses the critical contributions and limitations of this artwork, leading to an evaluation of Wolfson’s conceptual performance as a whole

GenAssist: Making Image Generation Accessible

Blind and low vision (BLV) creators use images to communicate with sighted audiences. However, creating or retrieving images is challenging for BLV creators as it is difficult to use authoring tools or assess image search results. Thus, creators limit the types of images they create or recruit sighted collaborators. While text-to-image generation models let creators generate high-fidelity images based on a text description (i.e. prompt), it is difficult to assess the content and quality of generated images. We present GenAssist, a system to make text-to-image generation accessible. Using our interface, creators can verify whether generated image candidates followed the prompt, access additional details in the image not specified in the prompt, and skim a summary of similarities and differences between image candidates. To power the interface, GenAssist uses a large language model to generate visual questions, vision-language models to extract answers, and a large language model to summarize the results. Our study with 12 BLV creators demonstrated that GenAssist enables and simplifies the process of image selection and generation, making visual authoring more accessible to all.Comment: For accessibility tagged pdf, please refer to the ancillary fil

A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

The development of large language models (LLMs) such as ChatGPT has brought a lot of attention recently. However, their evaluation in the benchmark academic datasets remains under-explored due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. In this paper, we aim to present a thorough evaluation of ChatGPT's performance on diverse academic datasets, covering tasks like question-answering, text summarization, code generation, commonsense reasoning, mathematical problem-solving, machine translation, bias detection, and ethical considerations. Specifically, we evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in these datasets. This makes our work the largest evaluation of ChatGPT in NLP benchmarks. In short, our study aims to validate the strengths and weaknesses of ChatGPT in various tasks and provide insights for future research using LLMs. We also report a new emergent ability to follow multi-query instructions that we mostly found in ChatGPT and other instruction-tuned models. Our extensive evaluation shows that even though ChatGPT is capable of performing a wide variety of tasks, and may obtain impressive performance in several benchmark datasets, it is still far from achieving the ability to reliably solve many challenging tasks. By providing a thorough assessment of ChatGPT's performance across diverse NLP tasks, this paper sets the stage for a targeted deployment of ChatGPT-like LLMs in real-world applications.Comment: Accepted by ACL 2023 Findings. The first three authors contributed equall

Arabic and English Loan Words in Bahasa: Implications for Foreign Language Pedagogy

Many Arab students are currently pursuing their education at Malaysian institutions, and they have to study Bahasa Malaysia as a university requirement to be able to communicate with people in the local community. Therefore, this study aims to help Arab students learn Bahasa easily as Bahasa contains many loan words from Arabic and English. This article gives Arab students examples of Arabic and English loan words with which they are familiar and shows them the different phonological adaptations of Arabic and English loan words in Bahasa as the Arabic, English and Bahasa sound systems are different. A corpus of non-technical Malay words that are commonly encountered in public places in Malaysia was collected and analyzed.  A contrastive analysis of Arabic and Malay, and English and Malay phonological systems was performed. Different homogenization processes are applied to Arabic and English loan words depending on the differences between Arabic and Malay, and English and Malay. Examples of Arabic loan words in Malay are: menara, Sabtu, Ahad, Akhir, tahniya. Examples of English loan words in Bahasa are: stesen, kelab, tren, kompleks, imigresen, destinasi. Further implications for learning Bahasa Malaysia by Arab students are given

Two Decades of Colorization and Decolorization for Images and Videos

Colorization is a computer-aided process, which aims to give color to a gray image or video. It can be used to enhance black-and-white images, including black-and-white photos, old-fashioned films, and scientific imaging results. On the contrary, decolorization is to convert a color image or video into a grayscale one. A grayscale image or video refers to an image or video with only brightness information without color information. It is the basis of some downstream image processing applications such as pattern recognition, image segmentation, and image enhancement. Different from image decolorization, video decolorization should not only consider the image contrast preservation in each video frame, but also respect the temporal and spatial consistency between video frames. Researchers were devoted to develop decolorization methods by balancing spatial-temporal consistency and algorithm efficiency. With the prevalance of the digital cameras and mobile phones, image and video colorization and decolorization have been paid more and more attention by researchers. This paper gives an overview of the progress of image and video colorization and decolorization methods in the last two decades.Comment: 12 pages, 19 figure