GestureGPT: Zero-shot Interactive Gesture Understanding and Grounding with Large Language Model Agents

Current gesture recognition systems primarily focus on identifying gestures within a predefined set, leaving a gap in connecting these gestures to interactive GUI elements or system functions (e.g., linking a 'thumb-up' gesture to a 'like' button). We introduce GestureGPT, a novel zero-shot gesture understanding and grounding framework leveraging large language models (LLMs). Gesture descriptions are formulated based on hand landmark coordinates from gesture videos and fed into our dual-agent dialogue system. A gesture agent deciphers these descriptions and queries about the interaction context (e.g., interface, history, gaze data), which a context agent organizes and provides. Following iterative exchanges, the gesture agent discerns user intent, grounding it to an interactive function. We validated the gesture description module using public first-view and third-view gesture datasets and tested the whole system in two real-world settings: video streaming and smart home IoT control. The highest zero-shot Top-5 grounding accuracies are 80.11% for video streaming and 90.78% for smart home tasks, showing potential of the new gesture understanding paradigm

Solution for SMART-101 Challenge of ICCV Multi-modal Algorithmic Reasoning Task 2023

In this paper, we present our solution to a Multi-modal Algorithmic Reasoning Task: SMART-101 Challenge. Different from the traditional visual question-answering datasets, this challenge evaluates the abstraction, deduction, and generalization abilities of neural networks in solving visuolinguistic puzzles designed specifically for children in the 6-8 age group. We employed a divide-and-conquer approach. At the data level, inspired by the challenge paper, we categorized the whole questions into eight types and utilized the llama-2-chat model to directly generate the type for each question in a zero-shot manner. Additionally, we trained a yolov7 model on the icon45 dataset for object detection and combined it with the OCR method to recognize and locate objects and text within the images. At the model level, we utilized the BLIP-2 model and added eight adapters to the image encoder VIT-G to adaptively extract visual features for different question types. We fed the pre-constructed question templates as input and generated answers using the flan-t5-xxl decoder. Under the puzzle splits configuration, we achieved an accuracy score of 26.5 on the validation set and 24.30 on the private test set

Innate Immune Cells: A Potential and Promising Cell Population for Treating Osteosarcoma

Advanced, recurrent, or metastasized osteosarcomas remain challenging to cure or even alleviate. Therefore, the development of novel therapeutic strategies is urgently needed. Cancer immunotherapy has greatly improved in recent years, with options including adoptive cellular therapy, vaccination, and checkpoint inhibitors. As such, immunotherapy is becoming a potential strategy for the treatment of osteosarcoma. Innate immunocytes, the first line of defense in the immune system and the bridge to adaptive immunity, are one of the vital effector cell subpopulations in cancer immunotherapy. Innate immune cell-based therapy has shown potent antitumor activity against hematologic malignancies and some solid tumors, including osteosarcoma. Importantly, some immune checkpoints are expressed on both innate and adaptive immune cells, modulating their functions in tumor immunity. Therefore, blocking or activating immune checkpoint-mediated downstream signaling pathways can improve the therapeutic effects of innate immune cell-based therapy. In this review, we summarize the current status and future prospects of innate immune cell-based therapy for the treatment of osteosarcoma, with a focus on the potential synergistic effects of combination therapy involving innate immunotherapy and immune checkpoint inhibitors/oncolytic viruses

Significance of cuproptosis- related genes in the diagnosis and classification of psoriasis

Cuproptosis is a novel form of cell death linked to mitochondrial metabolism and is mediated by protein lipoylation. The mechanism of cuproptosis in many diseases, such as psoriasis, remains unclear. In this study, signature diagnostic markers of cuproptosis were screened by differential analysis between psoriatic and non-psoriatic patients. The differentially expressed cuproptosis-related genes (CRGs) for patients with psoriasis were screened using the GSE178197 dataset from the gene expression omnibus database. The biological roles of CRGs were identified by GO and KEGG enrichment analyses, and the candidates of cuproptosis-related regulators were selected from a nomogram model. The consensus clustering approach was used to classify psoriasis into clusters and the principal component analysis algorithms were constructed to calculate the cuproptosis score. Finally, latent diagnostic markers and drug sensitivity were analyzed using the pRRophetic R package. The differential analysis revealed that CRGs (MTF1, ATP7B, and SLC31A1) are significantly expressed in psoriatic patients. GO and KEGG enrichment analyses showed that the biological functions of CRGs were mainly related to acetyl-CoA metabolic processes, the mitochondrial matrix, and acyltransferase activity. Compared to the machine learning method used, the random forest model has higher accuracy in the occurrence of cuproptosis. However, the decision curve of the candidate cuproptosis regulators analysis showed that patients can benefit from the nomogram model. The consensus clustering analysis showed that psoriasis can be grouped into three patterns of cuproptosis (clusterA, clusterB, and clusterC) based on selected important regulators of cuproptosis. In advance, we analyzed the immune characteristics of patients and found that clusterA was associated with T cells, clusterB with neutrophil cells, and clusterC predominantly with B cells. Drug sensitivity analysis showed that three cuproptosis regulators (ATP7B, SLC31A1, and MTF1) were associated with the drug sensitivity. This study provides insight into the specific biological functions and related mechanisms of CRGs in the development of psoriasis and indicates that cuproptosis plays a non-negligible role. These results may help guide future treatment strategies for psoriasis

Orphan Nuclear Receptor Nur77 Inhibits Cardiac Hypertrophic Response to Beta-Adrenergic Stimulation.

The orphan nuclear receptor Nur77 plays critical roles in cardiovascular diseases, and its expression is markedly induced in the heart after beta-adrenergic receptor (β-AR) activation. However, the functional significance of Nur77 in β-AR signaling in the heart remains unclear. By using Northern blot, Western blot, and immunofluorescent staining assays, we showed that Nur77 expression was markedly upregulated in cardiomyocytes in response to multiple hypertrophic stimuli, including isoproterenol (ISO), phenylephrine (PE), and endothelin-1 (ET-1). In a time- and dose-dependent manner, ISO increases Nur77 expression in the nuclei of cardiomyocytes. Overexpression of Nur77 markedly inhibited ISO-induced cardiac hypertrophy by inducing nuclear translocation of Nur77 in cardiomyocytes. Furthermore, cardiac overexpression of Nur77 by intramyocardial injection of Ad-Nur77 substantially inhibited cardiac hypertrophy and ameliorated cardiac dysfunction after chronic infusion of ISO in mice. Mechanistically, we demonstrated that Nur77 functionally interacts with NFATc3 and GATA4 and inhibits their transcriptional activities, which are critical for the development of cardiac hypertrophy. These results demonstrate for the first time that Nur77 is a novel negative regulator for the β-AR-induced cardiac hypertrophy through inhibiting the NFATc3 and GATA4 transcriptional pathways. Targeting Nur77 may represent a potentially novel therapeutic strategy for preventing cardiac hypertrophy and heart failure

A comparative analysis of near-infrared image colorization methods for low-power NVIDIA Jetson embedded systems

The near-infrared (NIR) image obtained by an NIR camera is a grayscale image that is inconsistent with the human visual spectrum. It can be difficult to perceive the details of a scene from an NIR scene; thus, a method is required to convert them to visible images, providing color and texture information. In addition, a camera produces so much video data that it increases the pressure on the cloud server. Image processing can be done on an edge device, but the computing resources of edge devices are limited, and their power consumption constraints need to be considered. Graphics Processing Unit (GPU)-based NVIDIA Jetson embedded systems offer a considerable advantage over Central Processing Unit (CPU)-based embedded devices in inference speed. For this study, we designed an evaluation system that uses image quality, resource occupancy, and energy consumption metrics to verify the performance of different NIR image colorization methods on low-power NVIDIA Jetson embedded systems for practical applications. The performance of 11 image colorization methods on NIR image datasets was tested on three different configurations of NVIDIA Jetson boards. The experimental results indicate that the Pix2Pix method performs best, with a rate of 27 frames per second on the Jetson Xavier NX. This performance is sufficient to meet the requirements of real-time NIR image colorization