Differentially Private Diffusion Models

While modern machine learning models rely on increasingly large training datasets, data is often limited in privacy-sensitive domains. Generative models trained with differential privacy (DP) on sensitive data can sidestep this challenge, providing access to synthetic data instead. We build on the recent success of diffusion models (DMs) and introduce Differentially Private Diffusion Models (DPDMs), which enforce privacy using differentially private stochastic gradient descent (DP-SGD). We investigate the DM parameterization and the sampling algorithm, which turn out to be crucial ingredients in DPDMs, and propose noise multiplicity, a powerful modification of DP-SGD tailored to the training of DMs. We validate our novel DPDMs on image generation benchmarks and achieve state-of-the-art performance in all experiments. Moreover, on standard benchmarks, classifiers trained on DPDM-generated synthetic data perform on par with task-specific DP-SGD-trained classifiers, which has not been demonstrated before for DP generative models. Project page and code: https://nv-tlabs.github.io/DPDM.Comment: Accepted at TMLR (https://openreview.net/forum?id=ZPpQk7FJXF

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

We present TexFusion (Texture Diffusion), a new method to synthesize textures for given 3D geometries, using large-scale text-guided image diffusion models. In contrast to recent works that leverage 2D text-to-image diffusion models to distill 3D objects using a slow and fragile optimization process, TexFusion introduces a new 3D-consistent generation technique specifically designed for texture synthesis that employs regular diffusion model sampling on different 2D rendered views. Specifically, we leverage latent diffusion models, apply the diffusion model's denoiser on a set of 2D renders of the 3D object, and aggregate the different denoising predictions on a shared latent texture map. Final output RGB textures are produced by optimizing an intermediate neural color field on the decodings of 2D renders of the latent texture. We thoroughly validate TexFusion and show that we can efficiently generate diverse, high quality and globally coherent textures. We achieve state-of-the-art text-guided texture synthesis performance using only image diffusion models, while avoiding the pitfalls of previous distillation-based methods. The text-conditioning offers detailed control and we also do not rely on any ground truth 3D textures for training. This makes our method versatile and applicable to a broad range of geometry and texture types. We hope that TexFusion will advance AI-based texturing of 3D assets for applications in virtual reality, game design, simulation, and more.Comment: Videos and more results on https://research.nvidia.com/labs/toronto-ai/texfusion

Advanced synaptic transistor device towards AI application in hardware perspective

For the past decades, the synaptic devices for the inmemory computing have been widely investigated due to the high-efficiency computing potential and the ability to mimic biological neurobehavior. However, the conventional twoterminal synaptic memristors show drawbacks of resistance reduction caused by large-scale paralleling and asynchronous storage-reading process, which limit its development. Recently, researchers have paid attention to the transistor-like artificial synapse. Due to the existence of insulator layer and the separation of input and read terminals, the three-terminal synaptic transistors are believed to have greater potential towards artificial intelligence (AI) application fields. In this work, a summary of recent progresses and the future challenges of synaptic transistors are discussed

Neuromorphic visual artificial synapse in-memory computing systems based on GeOx-coated MXene nanosheets

Artificial synapses with light signal perception capability offer the ability to neuromorphic visual signal processing system on demand. In light of the excellent optical and electrical characteristics, the low-dimensional materials have become one of the most favorable candidates of the key component for optoelectronic artificial synapses. Previously, our group originally proposed the synthesis of germanium oxide-coated MXene nanosheets. In this work, we further applied this technology into the optoelectronic synaptic thin-film transistors for the first time. The devices exhibited the adjustable postsynaptic current behaviors under the visible light inputs. Moreover, the potentiation and depression operation modes of the devices further improved the application potential of the devices in mimicking biological synapses. Regulated by the wavelength of incident lights, the proposed artificial synapse could effectively help detect the target area of the image. Eventually, we further showed the results of the devices in the projects of neural network computing task. The long-term potentiation/depression characteristics of the conductance were applied to the synaptic weight matrix for image identification and path recognition tasks. By adding knowledge transfer in the process of recognition, the epoch required for convergence has been greatly reduced. The result of high noise tolerance revealed the great potential of the proposed transistors in establishing high-efficiency and robustness hardware neuromorphic systems for in-memory computing

Egocentric video: a new tool for capturing hand use of individuals with spinal cord injury at home

Abstract Background Current upper extremity outcome measures for persons with cervical spinal cord injury (cSCI) lack the ability to directly collect quantitative information in home and community environments. A wearable first-person (egocentric) camera system is presented that aims to monitor functional hand use outside of clinical settings. Methods The system is based on computer vision algorithms that detect the hand, segment the hand outline, distinguish the user’s left or right hand, and detect functional interactions of the hand with objects during activities of daily living. The algorithm was evaluated using egocentric video recordings from 9 participants with cSCI, obtained in a home simulation laboratory. The system produces a binary hand-object interaction decision for each video frame, based on features reflecting motion cues of the hand, hand shape and colour characteristics of the scene. Results The output from the algorithm was compared with a manual labelling of the video, yielding F1-scores of 0.74 ± 0.15 for the left hand and 0.73 ± 0.15 for the right hand. From the resulting frame-by-frame binary data, functional hand use measures were extracted: the amount of total interaction as a percentage of testing time, the average duration of interactions in seconds, and the number of interactions per hour. Moderate and significant correlations were found when comparing these output measures to the results of the manual labelling, with ρ = 0.40, 0.54 and 0.55 respectively. Conclusions These results demonstrate the potential of a wearable egocentric camera for capturing quantitative measures of hand use at home

Advanced artificial synaptic thin-film transistor based on doped potassium ions for neuromorphic computing via third-generation neural network

As the basic and essential unit of neuromorphic computing systems, artificial synaptic devices have great potential to accelerate high-performance parallel computation, artificial intelligence, and adaptive learning. Among the proposed artificial synaptic devices, the synaptic transistors are well considered to be one of the most suitable devices for simulating artificial intelligence. So far, synaptic transistors based on iontronic have been proposed and proved to demonstrate great potential in artificial intelligence applications. However, little research specifically focused on improving the device's ability to mimic synaptic behaviour. Here, we proposed the enhancement of synaptic properties of the solution-based thin-film transistors based on potassium ion conduction in the dielectric layer for the first time. Due to the formation of a gated electrical double-layer, the transistor exhibited an enlarged memory window. Based on this, the excitatory postsynaptic current in the synaptic thin-film transistor was modified accordingly, which further enhanced the suitability of the proposed synaptic thin-film transistor for simulating biological synapses. In addition, considerable synaptic properties were evaluated elaborately, including paired-pulse facilitation, short-term memory, long-term memory, and spike-time-dependent-plasticity. Most importantly, according to the impressive results of the Artificial Neural Network algorithm's image recognition simulation, the simulation image recognition rate based on the mentioned artificial synaptic devices reached as high as 92%. Last but not least, in order to simulate biological neurobehavior more closely, the Spiking Neural Network algorithm was also successfully implemented to complete the specified machine learning task, which further proved the great potential of the synaptic devices in advanced low-power neural network systems

Brain-like optoelectronic artificial synapses with ultralow energy consumption based on MXene floating-gates for emotion recognition

In the new generation of brain-like optoelectronic visual signal processing and artificial perception systems, floating-gate artificial synaptic devices based on two-dimensional materials represent a feasible route.</jats:p

Recent Advances in Two-Dimensional Magnets: Physics and Devices towards Spintronic Applications

The emergence of low-dimensional nanomaterials has brought revolutionized development of magnetism, as the size effect can significantly influence the spin arrangement. Since the first demonstration of truly two-dimensional magnetic materials (2DMMs) in 2017, a wide variety of magnetic phases and associated properties have been exhibited in these 2DMMs, which offer a new opportunity to manipulate the spin-based devices efficiently in the future. Herein, we focus on the recent progress of 2DMMs and heterostructures in the aspects of their structural characteristics, physical properties, and spintronic applications. Firstly, the microscopy characterization of the spatial arrangement of spins in 2D lattices is reviewed. Afterwards, the optical probes in the light-matter-spin interactions at the 2D scale are discussed. Then, particularly, we systematically summarize the recent work on the electronic and spintronic devices of 2DMMs. In the section of electronic properties, we raise several exciting phenomena in 2DMMs, i.e., long-distance magnon transport, field-effect transistors, varying magnetoresistance behavior, and (quantum) anomalous Hall effect. In the section of spintronic applications, we highlight spintronic devices based on 2DMMs, e.g., spin valves, spin-orbit torque, spin field-effect transistors, spin tunneling field-effect transistors, and spin-filter magnetic tunnel junctions. At last, we also provide our perspectives on the current challenges and future expectations in this field, which may be a helpful guide for theorists and experimentalists who are exploring the optical, electronic, and spintronic properties of 2DMMs