Generating Visual Scenes from Touch

An emerging line of work has sought to generate plausible imagery from touch. Existing approaches, however, tackle only narrow aspects of the visuo-tactile synthesis problem, and lag significantly behind the quality of cross-modal synthesis methods in other domains. We draw on recent advances in latent diffusion to create a model for synthesizing images from tactile signals (and vice versa) and apply it to a number of visuo-tactile synthesis tasks. Using this model, we significantly outperform prior work on the tactile-driven stylization problem, i.e., manipulating an image to match a touch signal, and we are the first to successfully generate images from touch without additional sources of information about the scene. We also successfully use our model to address two novel synthesis problems: generating images that do not contain the touch sensor or the hand holding it, and estimating an image's shading from its reflectance and touch.Comment: ICCV 2023; Project site: https://fredfyyang.github.io/vision-from-touch

Nanosheet-Assembled ZnO Microflower Photocatalysts

Large scale ZnO microflowers assembled by numerous nanosheets are synthesized through a facile and effective hydrothermal route. The structure and morphology of the resultant products are characterized by X-ray diffraction (XRD) and scanning electron microscope (SEM). Photocatalytic properties of the as-synthesized products are also investigated. The results demonstrate that eosin red aqueous solution can be degraded over 97% after 110 min under UV light irradiation. In addition, methyl orange (MO) and Congo red (CR) aqueous solution degradation experiments also are conducted in the same condition, respectively. It showed that nanosheet-assembled ZnO microflowers represent high photocatalytic activities with a degradation efficiency of 91% for CR with 90 min of irradiation and 90% for MO with 60 min of irradiation. The reported ZnO products may be promising candidates as the photocatalysts in waste water treatment

A Topology-Controlled Photonic Cavity Based on the Near-Conservation of the Valley Degree of Freedom

We demonstrate a novel path to localizing topologically-nontrivial photonic edge modes along their propagation direction. Our approach is based on the near-conservation of the photonic valley degree of freedom associated with valley-polarized edge states. When the edge state is reflected from a judiciously oriented mirror, its optical energy is localized at the mirror surface because of an extended time delay required for valley-index-flipping. The degree of energy localization at the resulting topology-controlled photonic cavity (TCPC) is determined by the valley-flipping time, which is in turn controlled by the geometry of the mirror. Intuitive analytic descriptions of the "leaky" and closed TCPCs are presented, and two specific designs--one for the microwave and the other for the optical spectral ranges--are proposed.Comment: 5 pages, 6 figure

Using personally controlled air movement to improve comfort after simulated summer commute

People often feel uncomfortably warm and sweaty in their workspace after commuting there by walking or cycling in summer. This is because body heat stored during the commute takes a substantial time to dissipate. People complaining about this uncomfortable transition may cause operators to lower the thermostat setpoint, causing long-term overcooling and wasting energy. In addition, space cooling is slow, requiring minutes to take effect. This study addresses how to improve comfort in the transition by increasing the availability of convective cooling, where the response time is in seconds. Thirty-five subjects (17 men and 18 women) dressed in 0.6 clo en-tered a test room after exercising at 4.4 met for 15 min in 30 ºC. The exercise emulates the commute activity in summer. The test room was controlled to 24, 26, and 28 ºC, with and without the option of cooling using fan-produced horizontal airflow. Subjects were sedentary for 60 minutes, during which subjective thermal responses and physiological responses were measured. The enhanced convective and evaporative heat loss caused by fans significantly shortened the time needed to reach thermal comfort after the exercise-induced thermal stress and improved the final comfort level. Compared to a typical indoor condition of 24 ºC and still air, 26 and 28 ºC with fans provided equal or better comfort more quickly, and inherently required much less energy to do so. Our study suggests that personally controlled air movement should be available in spaces where thermal and metabolic down-steps take plac

Dance with You: The Diversity Controllable Dancer Generation via Diffusion Models

Recently, digital humans for interpersonal interaction in virtual environments have gained significant attention. In this paper, we introduce a novel multi-dancer synthesis task called partner dancer generation, which involves synthesizing virtual human dancers capable of performing dance with users. The task aims to control the pose diversity between the lead dancer and the partner dancer. The core of this task is to ensure the controllable diversity of the generated partner dancer while maintaining temporal coordination with the lead dancer. This scenario varies from earlier research in generating dance motions driven by music, as our emphasis is on automatically designing partner dancer postures according to pre-defined diversity, the pose of lead dancer, as well as the accompanying tunes. To achieve this objective, we propose a three-stage framework called Dance-with-You (DanY). Initially, we employ a 3D Pose Collection stage to collect a wide range of basic dance poses as references for motion generation. Then, we introduce a hyper-parameter that coordinates the similarity between dancers by masking poses to prevent the generation of sequences that are over-diverse or consistent. To avoid the rigidity of movements, we design a Dance Pre-generated stage to pre-generate these masked poses instead of filling them with zeros. After that, a Dance Motion Transfer stage is adopted with leader sequences and music, in which a multi-conditional sampling formula is rewritten to transfer the pre-generated poses into a sequence with a partner style. In practice, to address the lack of multi-person datasets, we introduce AIST-M, a new dataset for partner dancer generation, which is publicly availiable. Comprehensive evaluations on our AIST-M dataset demonstrate that the proposed DanY can synthesize satisfactory partner dancer results with controllable diversity.Comment: Accepted by ACM MM 202

Touch and Go: Learning from Human-Collected Vision and Touch

The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world. We propose a dataset with paired visual and tactile data called Touch and Go, in which human data collectors probe objects in natural environments using tactile sensors, while simultaneously recording egocentric video. In contrast to previous efforts, which have largely been confined to lab settings or simulated environments, our dataset spans a large number of "in the wild" objects and scenes. To demonstrate our dataset's effectiveness, we successfully apply it to a variety of tasks: 1) self-supervised visuo-tactile feature learning, 2) tactile-driven image stylization, i.e., making the visual appearance of an object more consistent with a given tactile signal, and 3) predicting future frames of a tactile signal from visuo-tactile inputs.Comment: Accepted by NeurIPS 2022 Track of Datasets and Benchmark