Array-Based Protein Sensing Using an Aggregation-Induced Emission (AIE) Light-Up Probe

Protein detection and identification are important for the diagnosis of diseases; however, the development of facile sensing probes still remains challenging. Here, we present an array-based "turn on" protein-sensing platform capable of detecting and identifying proteins using aggregation-induced emission luminogens (AIEgens). The water-soluble AIEgens in which fluorescence was initially turned off showed strong fluorescence in the presence of nanomolar concentrations of proteins via restriction of the intramolecular rotation of the AIEgens. The binding affinities between the AIEgens and proteins were associated with various chemical functional groups on AIEgens, resulting in distinct fluorescent-signal outcomes for each protein. The combined fluorescence outputs provided sufficient information to detect and discriminate proteins of interest by linear discriminant analysis. Furthermore, the array-based sensor enabled classification of different concentrations of specific proteins. These results provide novel insight into the use of the AIEgens as a new type of sensing probe in array-based systems

LISA: Localized Image Stylization with Audio via Implicit Neural Representation

We present a novel framework, Localized Image Stylization with Audio (LISA) which performs audio-driven localized image stylization. Sound often provides information about the specific context of the scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. In this work, we propose a framework that a user provides an audio input to localize the sound source in the input image and another for locally stylizing the target object or scene. LISA first produces a delicate localization map with an audio-visual localization network by leveraging CLIP embedding space. We then utilize implicit neural representation (INR) along with the predicted localization map to stylize the target object or scene based on sound information. The proposed INR can manipulate the localized pixel values to be semantically consistent with the provided audio input. Through a series of experiments, we show that the proposed framework outperforms the other audio-guided stylization methods. Moreover, LISA constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/Comment: ICCV202

Sound-Guided Semantic Video Generation

The recent success in StyleGAN demonstrates that pre-trained StyleGAN latent space is useful for realistic video generation. However, the generated motion in the video is usually not semantically meaningful due to the difficulty of determining the direction and magnitude in the StyleGAN latent space. In this paper, we propose a framework to generate realistic videos by leveraging multimodal (sound-image-text) embedding space. As sound provides the temporal contexts of the scene, our framework learns to generate a video that is semantically consistent with sound. First, our sound inversion module maps the audio directly into the StyleGAN latent space. We then incorporate the CLIP-based multimodal embedding space to further provide the audio-visual relationships. Finally, the proposed frame generator learns to find the trajectory in the latent space which is coherent with the corresponding sound and generates a video in a hierarchical manner. We provide the new high-resolution landscape video dataset (audio-visual pair) for the sound-guided video generation task. The experiments show that our model outperforms the state-of-the-art methods in terms of video quality. We further show several applications including image and video editing to verify the effectiveness of our method

Analysis of the Activity and Travel Patterns of the Elderly Using Mobile Phone-Based Hourly Locational Trajectory Data: Case Study of Gangnam, Korea

Rapid demographic ageing is a global challenge and has tremendous implications for transportation planning, because the mobility of elderly people is an essential element for active ageing. Although many studies have been conducted on this issue, most of them have been focused on aggregated travel patterns of the elderly, limited in spatiotemporal analysis, and most importantly primarily relied on sampled (2–3%) household travel surveys, omitting some trips and having concerns of quality and credibility. The objectives of this study are to present more in-depth analysis of the elderly’s spatiotemporal activity and travel behaviors, to compare them with other age and gender groups, and to draw implications for sustainable transportation for the elderly. For our analysis, we used locational trajectory-based mobile phone data in Gangnam, Korea. The data differs from sampled household travel survey data, as mobile phone data represents the entire population and can capture comprehensive travelers’ movements, including peculiarities. Consistent with previous researches, the results of this study showed that there were differences in activity and travel patterns between age and gender groups. However, some different results were obtained as well: for instance, the average nonhome activity time per person for the elderly was shorter than that of the nonelderly, but the average numbers of nonhome activities and trips were rather higher than those of nonelderly people. The results of this study and advantage of using mobile phone data will help policymakers understand the activities and movements of the elderly and prepare future sustainable transportation

A Stochastic Optimization Model for Sustainable Multimodal Transportation for Bioenergy Production

While many previous studies have suggested well-defined procedures to find appropriate supply chains, a limited number of studies have been conducted with uncertain values relating to transportation costs. Most of these have included only limited detail on multimodal transportation, or have not considered economic, social, and environmental transportation cost factors together. The main purpose of this study is to suggest a multi-objective stochastic model for sustainable biomass transportation, and to identify the impact level of model selection on the transportation mode. It begins with a deterministic formulation of sustainable transportation, which is then modified to a stochastic problem with vectorization of cost parameters. Based on the model developed, we examined four uncertainty cases from a combination of annual capacity and average distance of biomass transportation. The experimental results provide more cost savings from multimodal transportation, which can be identified if we analyze transportation costs with stochastic modeling. Regarding short-distance plant cases, the study reveals that the impact of the utilization of stochastic methods is insignificant, as the costs savings from multimodal transportation is trivial. Other findings from the experiments show that multimodal transportation could provide cost savings in the economic cost factor, except in the case of low annual capacity and short average distance

DNN-Based Forensic Watermark Tracking System for Realistic Content Copyright Protection

The metaverse-related content market is active and the demand for immersive content is increasing. However, there is no definition for granting copyrights to the content produced using artificial intelligence and discussions are still ongoing. We expect that the need for copyright protection for immersive content used in the metaverse environment will emerge and that related copyright protection techniques will be required. In this paper, we present the idea of 3D-to-2D watermarking so that content creators can protect the copyright of immersive content available in the metaverse environment. We propose an immersive content copyright protection using a deep neural network (DNN), a neural network composed of multiple hidden layers, and a forensic watermark

Array-based Protein Sensing using an Aggregation Induced Emission Light-Up Probe

Protein detection and identification is important for diagnosis of diseases but the development of facile sensing probes still remains challenging. Herein, we present an array-based ???turn on??? protein sensing platform that can detect and identify proteins using aggregation induced emission luminogens (AIEgens). The water-soluble AIEgens that the fluorescence are initially turned-off, showed strong fluorescence in the presence of nanomolar concentration of proteins via the restriction of the intramolecular rotation (RIR) of the AIEgens. The binding affinities between the AIEgens and proteins were associated with various chemical functional groups on AIEgens, resulting in the distinct fluorescent signal outcomes for each protein. The combining fluorescence outputs provided sufficient information to detect and discriminate proteins of interest by linear discriminant analysis (LDA). Furthermore, the array-based sensor could be used to classify different concentration of specific proteins. This study provides a new sight into the use of the AIEgens as a new type of sensing probe installable in an array-based system