GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest

Instruction tuning large language model (LLM) on image-text pairs has achieved unprecedented vision-language multimodal abilities. However, their vision-language alignments are only built on image-level, the lack of region-level alignment limits their advancements to fine-grained multimodal understanding. In this paper, we propose instruction tuning on region-of-interest. The key design is to reformulate the bounding box as the format of spatial instruction. The interleaved sequences of visual features extracted by the spatial instruction and the language embedding are input to LLM, and trained on the transformed region-text data in instruction tuning format. Our region-level vision-language model, termed as GPT4RoI, brings brand new conversational and interactive experience beyond image-level understanding. (1) Controllability: Users can interact with our model by both language and spatial instructions to flexibly adjust the detail level of the question. (2) Capacities: Our model supports not only single-region spatial instruction but also multi-region. This unlocks more region-level multimodal capacities such as detailed region caption and complex region reasoning. (3) Composition: Any off-the-shelf object detector can be a spatial instruction provider so as to mine informative object attributes from our model, like color, shape, material, action, relation to other objects, etc. The code, data, and demo can be found at https://github.com/jshilong/GPT4RoI.Comment: Code has been released at https://github.com/jshilong/GPT4Ro

Modeling of nonlinear system based on deep learning framework

A novel modeling based on deep learning framework which can exactly manifest the characteristics of nonlinear system is proposed in this paper. Specifically, a Deep Reconstruction Model (DRM) is defined integrating with the advantages of the deep learning and Elman neural network (ENN). The parameters of the model are initialized by performing unsupervised pretraining in a layer-wise fashion using Restricted Boltzmann Machines (RBMs) to provide a faster convergence rate for modeling. ENN can be used to manifest the memory effect of system. To validate the proposed approach, two different nonlinear systems are used for experiments. The first one corresponds to the Class-D power amplifier (CDPA) which operates in the ohmic and cut-off regions. According to error of time domain and spectrum, Back Propagation Neural Network model improved by RBMs (BP-RBMs) and ENN are compared of different input signals which are the simulated two-tone signal and actual square wave signal. The second system is a permanent magnet synchronous motor (PMSM) servo control system based on fuzzy PID control strategy. In terms of simulated and actual speed curves, BP-RBMs, DRM and ENN model are adopted on comparison respectively. It is shown by experimental results that the proposed model with fewer parameters and iteration number can reconstruct the nonlinear system accurately, and depict the memory effect, the nonlinear distortion and the dynamic performance of system precisely.This work was supported in part by the Foundation of Key Laboratory of China’s Education Ministry and A Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.http://link.springer.com/journal/110712017-05-31hb2016Electrical, Electronic and Computer Engineerin

An improved time-frequency representation based on nonlinear mode decomposition and adaptive optimal kernel

Time-frequency representation (TFR) based on Adaptive Optimal Kernel (AOK) normally performs well only for monocomponent signals and has poor noise robustness. To overcome the shortcomings of AOK TFR mentioned above, a new TFR algorithm is proposed here by integrating nonlinear mode decomposition (NMD) with AOK TFR. NMD is used to decompose multicomponent signals into a bundle of meaningful oscillations and then AOK is applied to compute the TFR of individual oscillations, finally all these TFRs are summed together to generate one TFR. Through quantitative comparison with other TFR methods to both simulated and real signals, the superiority of proposed TFR based on NMD and AOK on removing noise and many other measurement index of TFR are shown.The Foundation of Key Laboratory of China’s Education Ministry (UASP1201) and A Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.http://www.eejournal.ktu.lt/index.php/eltam2016Electrical, Electronic and Computer Engineerin

Kinesin Is an Evolutionarily Fine-Tuned Molecular Ratchet-and-Pawl Device of Decisively Locked Direction

Conventional kinesin is a dimeric motor protein that transports membranous organelles toward the plus-end of microtubules (MTs). Individual kinesin dimers show steadfast directionality and hundreds of consecutive steps, yetthe detailed physical mechanism remains unclear. Here we compute free energies for the entire dimer-MT system for all possible interacting configurations by taking full account of molecular details. Employing merely first principles and several measured binding and barrier energies, the system-level analysis reveals insurmountable energy gaps between configurations, asymmetric ground state caused by mechanically lifted configurational degeneracy, and forbidden transitions ensuring coordination between both motor domains for alternating catalysis. This wealth of physical effects converts a kinesin dimer into a molecular ratchet-and-pawl device, which determinedly locks the dimer's movement into the MT plus-end and ensures consecutive steps in hand-over-hand gait.Under a certain range of extreme loads, however, the ratchet-and-pawl device becomes defective but not entirely abolished to allow consecutive back-steps. This study yielded quantitative evidence that kinesin's multiple molecular properties have been evolutionarily adapted to fine-tune the ratchet-and-pawl device so as to ensure the motor's distinguished performance.Comment: 10 printed page

Innate Immune Response to Viral Vectors in Gene Therapy

Viral vectors play a pivotal role in the field of gene therapy, with several related drugs having already gained clinical approval from the EMA and FDA. However, numerous viral gene therapy vectors are currently undergoing pre-clinical research or participating in clinical trials. Despite advancements, the innate response remains a significant barrier impeding the clinical development of viral gene therapy. The innate immune response to viral gene therapy vectors and transgenes is still an important reason hindering its clinical development. Extensive studies have demonstrated that different DNA and RNA sensors can detect adenoviruses, adeno-associated viruses, and lentiviruses, thereby activating various innate immune pathways such as Toll-like receptor (TLR), cyclic GMP-AMP synthase–stimulator of interferon genes (cGAS-STING), and retinoic acid-inducible gene I–mitochondrial antiviral signaling protein (RLR-MAVS). This review focuses on elucidating the mechanisms underlying the innate immune response induced by three widely utilized viral vectors: adenovirus, adeno-associated virus, and lentivirus, as well as the strategies employed to circumvent innate immunity

An MR-Based Viscosity-Type Regularization Method for Electrical Property Tomography

Here, a method based on viscosity-type regularization is proposed for magnetic resonance electrical property tomography (MREPT) to mitigate persistent artifacts when it is used to reconstruct a map of electrical properties based on data from a magnetic resonance imaging scanner. The challenges for solving the corresponding partial differential equation (PDE) are discussed in detail. The existing artifacts in the numerical results are pointed out and classified. The methods in the literature for MREPT are mainly based on an assumption of local homogeneity, which makes the approach simple but leads to artifacts in the transition region where electrical properties vary rapidly. Recent work has focused on eliminating the assumption of local homogeneity, and one of the solutions is convection–reaction MREPT that is based on a first-order PDE. Numerical solutions of the PDE have persistent artifacts in certain regions and global spurious oscillations. Here, a method based on viscosity-type regularization is proposed to effectively mitigate the aforementioned problems. Finite difference method is used for discretizing the governing PDE. Numerical experiments are presented to analyze the problem in detail. Electrical properties of different phantoms are successfully retrieved. The efficiency, accuracy, and noise tolerance of the proposed method are illustrated with numerical results