LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VIDComment: Code is available at https://github.com/dvlab-research/LLaMA-VI

Hierarchical Dense Correlation Distillation for Few-Shot Segmentation-Extended Abstract

Few-shot semantic segmentation (FSS) aims to form class-agnostic models segmenting unseen classes with only a handful of annotations. Previous methods limited to the semantic feature and prototype representation suffer from coarse segmentation granularity and train-set overfitting. In this work, we design Hierarchically Decoupled Matching Network (HDMNet) mining pixel-level support correlation based on the transformer architecture. The self-attention modules are used to assist in establishing hierarchical dense features, as a means to accomplish the cascade matching between query and support features. Moreover, we propose a matching module to reduce train-set overfitting and introduce correlation distillation leveraging semantic correspondence from coarse resolution to boost fine-grained segmentation. Our method performs decently in experiments. We achieve 50.0% mIoU on COCO dataset one-shot setting and 56.0% on five-shot segmentation, respectively. The code will be available on the project website. We hope our work can benefit broader industrial applications where novel classes with limited annotations are required to be decently identified.Comment: Accepted to CVPR 2023 VISION Workshop, Oral. The extended abstract of Hierarchical Dense Correlation Distillation for Few-Shot Segmentation. arXiv admin note: substantial text overlap with arXiv:2303.1465

Using Network Processor to Establish Security Agent for AODV Routing Protocol

Network Processor (NP) is optimized to perform special network functionalities. It has highly parallel processing architecture to achieve high performance. Ad hoc network is an exciting research aspect due to the characters of self-organization、 dynamically changing topology and temporary network life. However, all the characters make the security problem more serious. Denial-of-Service (DoS) attack is the main puzzle in the security of Ad hoc network. A novel NP-based security scheme is proposed to combat the attack in AODV routing protocol. Security agent is established by a hardware thread in NP. Agent can update itself at some interval by the trustworthiness of the neighbor nodes. Agent can trace the RREQ and RREP messages stream to aggregate the key information to link list and analyze them by intrusion detection algorithm. NS2 simulator is expanded to validate the security scheme. Simulation results show that NP-based security scheme is highly effective to detect and block DoS attack

Research hotspots and emerging trends of deep learning applications in orthopedics: A bibliometric and visualized study

BackgroundAs a research hotspot, deep learning has been continuously combined with various research fields in medicine. Recently, there is a growing amount of deep learning-based researches in orthopedics. This bibliometric analysis aimed to identify the hotspots of deep learning applications in orthopedics in recent years and infer future research trends.MethodsWe screened global publication on deep learning applications in orthopedics by accessing the Web of Science Core Collection. The articles and reviews were collected without language and time restrictions. Citespace was applied to conduct the bibliometric analysis of the publications.ResultsA total of 822 articles and reviews were finally retrieved. The analysis showed that the application of deep learning in orthopedics has great prospects for development based on the annual publications. The most prolific country is the USA, followed by China. University of California San Francisco, and Skeletal Radiology are the most prolific institution and journal, respectively. LeCun Y is the most frequently cited author, and Nature has the highest impact factor in the cited journals. The current hot keywords are convolutional neural network, classification, segmentation, diagnosis, image, fracture, and osteoarthritis. The burst keywords are risk factor, identification, localization, and surgery. The timeline viewer showed two recent research directions for bone tumors and osteoporosis.ConclusionPublications on deep learning applications in orthopedics have increased in recent years, with the USA being the most prolific. The current research mainly focused on classifying, diagnosing and risk predicting in osteoarthritis and fractures from medical images. Future research directions may put emphasis on reducing intraoperative risk, predicting the occurrence of postoperative complications, screening for osteoporosis, and identification and classification of bone tumors from conventional imaging

Marmoset Viral Hepatic Inflammation Induced by Hepatitis C Virus Core Protein via IL-32.

Common marmosets infected with GB virus-B (GBV-B) chimeras containing hepatitis C virus (HCV) core and envelope proteins (CE1E2p7) developed more severe hepatitis than those infected with HCV envelope proteins (E1E2p7), suggesting that HCV core protein might be involved in the pathogenesis of viral hepatitis. The potential role of HCV core in hepatic inflammation was investigated. Six individual cDNA libraries of liver tissues from HCV CE1E2p7 or E1E2p7 chimera-infected marmosets (three animals per group) were constructed and sequenced. By differential expression gene analysis, 30 of 632 mRNA transcripts were correlated with the immune system process, which might be associated with hepatitis. A protein-protein interaction network was constituted by STRING database based on these 30 differentially expressed genes (DEGs), showing that IL-32 might play a central regulatory role in HCV core-related hepatitis. To investigate the effect of HCV core protein on IL-32 production, HCV core expressing and mock constructs were transfected into Huh7 cells. IL-32 mRNA and secretion protein were detected at significantly higher levels in cells expressing HCV core protein than in those without HCV core expression (P < 0.01 and P < 0.001, respectively). By KEGG enrichment analysis and using the specific signaling pathway inhibitor LY294002 for inhibition of PI3K, IL-32 expression was significantly reduced (P < 0.001). In conclusion, HCV core protein induces an increase of IL-32 expression via the PI3K pathway in hepatic cells, which played a major role in development of HCV-related severe hepatitis

Detection of Brucellae in peripheral blood mononuclear cells for monitoring therapeutic efficacy of brucellosis infection.

BACKGROUND: Brucellosis is one of the most severe widespread zoonoses caused by the Gram-negative bacterium Brucella species. The diagnosis and clinical assessment of human brucellosis are very important for the management of patients, while there is a lack of effective methods to detect Brucellae. Classical culture of Brucella species is time consuming and often fails. A simple and sensitive assay is needed for diagnosis of Brucella infection and monitoring of treatment in man. METHODS: Blood samples and peripheral blood mononuclear cells (PBMCs) were collected from 154 patients hospitalized for brucellosis. Brucella antibodies were detected by Rose Bengal Plate Test (RBPT), Standard Tube Agglutination Test (SAT) and enzyme-linked immunosorbent assay (ELISA). Intracellular Brucellae were detected by blood culture and immunofluorescence staining (IFS). RESULTS: Among 154 brucellosis patients, 59.7% (92/154) were antibody reactive by RBPT, 81.8% (126/154) by SAT and 95.5% (147/154) by ELISA, respectively. Only 3.2% (5/154) of patient blood samples resulted in positive Brucella culture, while 68.8% (106/154) carried IFS detectable Brucella antigens in PBMCs. Gender (P = 0.01) but not age (P > 0.05) was a significant risk factor. The frequency of intracellular Brucella antigens was similar between patients receiving different treatment regimens (P > 0.05). However, a significant decrease of intracellular Brucellae was observed only in patients with acute brucellosis after the third course of treatment (P < 0.05), suggesting that current regimens to treat chronic brucellosis were not effective. CONCLUSIONS: IFS appears a sensitive assay for detection of Brucella antigens in PBMCs and could be used for diagnosis and therapeutic monitoring of brucellosis in clinical practice

Emerging Applications of Deep Learning in Bone Tumors: Current Advances and Challenges

Deep learning is a subfield of state-of-the-art artificial intelligence (AI) technology, and multiple deep learning-based AI models have been applied to musculoskeletal diseases. Deep learning has shown the capability to assist clinical diagnosis and prognosis prediction in a spectrum of musculoskeletal disorders, including fracture detection, cartilage and spinal lesions identification, and osteoarthritis severity assessment. Meanwhile, deep learning has also been extensively explored in diverse tumors such as prostate, breast, and lung cancers. Recently, the application of deep learning emerges in bone tumors. A growing number of deep learning models have demonstrated good performance in detection, segmentation, classification, volume calculation, grading, and assessment of tumor necrosis rate in primary and metastatic bone tumors based on both radiological (such as X-ray, CT, MRI, SPECT) and pathological images, implicating a potential for diagnosis assistance and prognosis prediction of deep learning in bone tumors. In this review, we first summarized the workflows of deep learning methods in medical images and the current applications of deep learning-based AI for diagnosis and prognosis prediction in bone tumors. Moreover, the current challenges in the implementation of the deep learning method and future perspectives in this field were extensively discussed

A high infectious simian adenovirus type 23 vector based vaccine efficiently protects common marmosets against Zika virus infection.

Zika virus (ZIKV) has spread in many countries or territories causing severe neurologic complications with potential fatal outcomes. The small primate common marmosets are susceptible to ZIKV, mimicking key features of human infection. Here, a novel simian adenovirus type 23 vector-based vaccine expressing ZIKV pre-membrane-envelope proteins (Sad23L-prM-E) was produced in high infectious titer. Due to determination of immunogenicity in mice, a single-dose of 3×108 PFU Sad23L-prM-E vaccine was intramuscularly inoculated to marmosets. This vaccine raised antibody titers of 104.07 E-specific and 103.13 neutralizing antibody (NAb), as well as robust specific IFN-γ secreting T-cell response (1,219 SFCs/106 cells) to E peptides. The vaccinated marmosets, upon challenge with a high dose of ZIKV (105 PFU) six weeks post prime immunization, reduced viremia by more than 100 folds, and the low level of detectable viral RNA (103.66) and T-cell response (>726 SFCs/106 PBMCs) were acquired 1-2 weeks post exposure to ZIKV, while non-vaccinated control marmosets developed long-term high titer of ZIKV (105.73 copies/ml) (P<0.05). No significant pathological lesions were observed in marmoset tissues. Sad23L-prM-E vaccine was detectable in spleen, liver and PBMCs at least 4 months post challenge. In conclusion, a prime immunization with Sad23L-prM-E vaccine was able to protect marmosets against ZIKV infection when exposed to a high dose of ZIKV. This Sad23L-prM-E vaccine is a promising vaccine candidate for prevention of ZIKV infection in humans