Adapting Language Models for Non-Parallel Author-Stylized Rewriting

Given the recent progress in language modeling using Transformer-based neural models and an active interest in generating stylized text, we present an approach to leverage the generalization capabilities of a language model to rewrite an input text in a target author's style. Our proposed approach adapts a pre-trained language model to generate author-stylized text by fine-tuning on the author-specific corpus using a denoising autoencoder (DAE) loss in a cascaded encoder-decoder framework. Optimizing over DAE loss allows our model to learn the nuances of an author's style without relying on parallel data, which has been a severe limitation of the previous related works in this space. To evaluate the efficacy of our approach, we propose a linguistically-motivated framework to quantify stylistic alignment of the generated text to the target author at lexical, syntactic and surface levels. The evaluation framework is both interpretable as it leads to several insights about the model, and self-contained as it does not rely on external classifiers, e.g. sentiment or formality classifiers. Qualitative and quantitative assessment indicates that the proposed approach rewrites the input text with better alignment to the target style while preserving the original content better than state-of-the-art baselines.Comment: Accepted for publication in Main Technical Track at AAAI 2

Controlling Styles in Neural Machine Translation with Activation Prompt

Controlling styles in neural machine translation (NMT) has attracted wide attention, as it is crucial for enhancing user experience. Earlier studies on this topic typically concentrate on regulating the level of formality and achieve some progress in this area. However, they still encounter two major challenges. The first is the difficulty in style evaluation. The style comprises various aspects such as lexis, syntax, and others that provide abundant information. Nevertheless, only formality has been thoroughly investigated. The second challenge involves excessive dependence on incremental adjustments, particularly when new styles are necessary. To address both challenges, this paper presents a new benchmark and approach. A multiway stylized machine translation (MSMT) benchmark is introduced, incorporating diverse categories of styles across four linguistic domains. Then, we propose a method named style activation prompt (StyleAP) by retrieving prompts from stylized monolingual corpus, which does not require extra fine-tuning. Experiments show that StyleAP could effectively control the style of translation and achieve remarkable performance.Comment: Accepted by Findings of ACL 2023; The code is available at https://github.com/IvanWang0730/StyleA

Deep Learning for Text Style Transfer: A Survey

Text style transfer is an important task in natural language generation, which aims to control certain attributes in the generated text, such as politeness, emotion, humor, and many others. It has a long history in the field of natural language processing, and recently has re-gained significant attention thanks to the promising performance brought by deep neural models. In this paper, we present a systematic survey of the research on neural text style transfer, spanning over 100 representative articles since the first neural text style transfer work in 2017. We discuss the task formulation, existing datasets and subtasks, evaluation, as well as the rich methodologies in the presence of parallel and non-parallel data. We also provide discussions on a variety of important topics regarding the future development of this task. Our curated paper list is at https://github.com/zhijing-jin/Text_Style_Transfer_SurveyComment: Computational Linguistics Journal 202

AUTHOR-SPECIFIC PREFIX-TUNING FOR PERSONALIZATION OF LARGE LANGUAGE MODELS

This article outlines a novel approach to the personalization of large language model (LLM) outputs to an individual writer in an artificial intelligence (AI)-assisted writing application without the need for fine-tuning or prompt-engineering. With this approach, an individual’s writing style may be encoded through a compact, learned model which maps writing samples to an author-embedding which may be prepended to the input of an LLM (in the manner of prefix-tuning) to steer the model to generate content in the writing style of that individual. The presented techniques involve several processing steps, including the selection of an optimal subset of an author’s writing samples, the training of an author embedding model, and the use of author-embeddings as a prefix to an LLM

Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training

In this work, we introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with 700 rephrased sentences and 1,000 sentences from the game Genshin Impact. While large language models (LLM) have shown promise in complex text style transfer, they have drawbacks such as data privacy concerns, network instability, and high deployment costs. To address these issues, we explore the effectiveness of small models (less than T5-3B) with implicit style pre-training through contrastive learning. We also propose a method for automated evaluation of text generation quality based on alignment with human evaluations using ChatGPT. Finally, we compare our approach with existing methods and show that our model achieves state-of-art performances of few-shot text style transfer models

A Call for Standardization and Validation of Text Style Transfer Evaluation

Text Style Transfer (TST) evaluation is, in practice, inconsistent. Therefore, we conduct a meta-analysis on human and automated TST evaluation and experimentation that thoroughly examines existing literature in the field. The meta-analysis reveals a substantial standardization gap in human and automated evaluation. In addition, we also find a validation gap: only few automated metrics have been validated using human experiments. To this end, we thoroughly scrutinize both the standardization and validation gap and reveal the resulting pitfalls. This work also paves the way to close the standardization and validation gap in TST evaluation by calling out requirements to be met by future research.Comment: Accepted to Findings of ACL 202

Text Style Transfer: A Review and Experimental Evaluation

The stylistic properties of text have intrigued computational linguistics researchers in recent years. Specifically, researchers have investigated the Text Style Transfer (TST) task, which aims to change the stylistic properties of the text while retaining its style independent content. Over the last few years, many novel TST algorithms have been developed, while the industry has leveraged these algorithms to enable exciting TST applications. The field of TST research has burgeoned because of this symbiosis. This article aims to provide a comprehensive review of recent research efforts on text style transfer. More concretely, we create a taxonomy to organize the TST models and provide a comprehensive summary of the state of the art. We review the existing evaluation methodologies for TST tasks and conduct a large-scale reproducibility study where we experimentally benchmark 19 state-of-the-art TST algorithms on two publicly available datasets. Finally, we expand on current trends and provide new perspectives on the new and exciting developments in the TST field

Rediscovering Cumulative Creativity From the Oral Formulaic Tradition to Digital Remix: Can I Get a Witness?, 13 J. Marshall Rev. Intell. Prop. L. 341 (2014)

For most of human history, the essential nature of creativity was understood to be cumulative and collective. This notion has been largely forgotten by modern policies that regulate creativity and speech. As hard as it may be to believe, the most valuable components of our immortal culture were created under a fully open regime with regard to access to pre-existing expressions and re-use. From the Platonic mimesis to Shakespeare’s “borrowed feathers,” the largest part of our culture has been produced under a paradigm in which imitation—even plagiarism—and social authorship formed constitutive elements of the creative moment. Pre-modern creativity spread from a continuous line of re-use and juxtaposition of pre-existing expressive content, transitioning from orality to textuality and then melding the two traditions. The cumulative and collaborative character of the oral-formulaic tradition dominated the development of epic literature. The literary pillars of Western culture, the Iliad and the Odyssey, were fully forged in the furnace of that tradition. Later, under the aegis of Macrobius’ art of rewriting and the Latin principles of imitatio, medieval epics grew out of similar dynamics of sharing and recombination of formulas and traditional patterns. Continuations, free re-use, and the re-modeling of iconic figures and characters, such as King Arthur and Roland, made chansons de geste and romance literature powerful vehicles in propelling cross-country circulation of culture. The parallelism between past and present highlights the incapacity of the present copyright system to recreate the cumulative and collaborative creative process that proved so fruitful in the past. In particular, the constant development and recursive use of iconic characters, which served as an engine for creativity in epic literature, is but a fading memory. This is because our policies for creativity are engineered in a fashion that stymies the re-use of information and knowledge, rather than facilitating it. Under the current regime, intellectual works are supposedly created as perfect, self-sustaining artifacts from the moment of their creation. Any modifications, derivations, and cumulative additions must secure preventive approval and must be paid off, as if they were nuisances to society. Rereading the history of aesthetics is particularly inspiring at the dawn of the networked age. The dynamics of sharing of pre-modern creativity parallel the features of digital networked creativity. As in the oral-formulaic tradition, digital creativity reconnects its exponential generative capacity to the ubiquity of participatory contributions. Additionally, the formula—the single unit to be used and re-used, worked and re-worked—is the building block of the remix culture as well as the oral formulaic tradition. Today, in an era of networked mass collaboration, ubiquitous online fan communities, user-based creativity, digital memes, and remix culture, the enclosure of knowledge brought about by an ever-expanding copyright paradigm is felt with renewed intensity. Therefore, I suggest that the communal, cumulative, social and collaborative nature of creativity and authorship should be rediscovered and should drive our policies. In order to plead my case, I have asked for the support of the most unexpected witnesses

Shakespeare and media ecology: beyond historicism and presentism

This article proposes media ecology-a combination of media studies and performance studies with literary and cultural history-as a research perspective for Shakespeare studies. In contrast to a hermeneutics of renewal-as evinced in both New Historicism and what has been called presentism-media ecology combines a sense of historical alterity with an awareness of the continuing transformations of Shakespeare in changing media settings: from manuscripts and printed texts to theatrical performances, music, opera, cinema, and new media. As an example, the article focuses on the masque in The Tempest, which poses obvious difficulties for a hermeneutics of renewal and is often cut from performance. Productions and adaptations frequently extend the spectacular qualities of the masque to The Tempest as a whole and ignore the skepticism about theatrical illusion that is voiced by Prospero in the play. In the case of The Tempest, cultural productions ranging from theatrical performances to the closing ceremony of the London Olympics of 20 12 are difficult to conceptualize in the framework of adaptation studies (which relies on the precedence of an original over its derivations). The article argues that media ecology can help scholars map out such connections and differences between performances and cultural phenomena relating to Shakespeare as cannot be fully grasped either in a historicist or presentist perspective