ATP: Adaptive Tensor Parallelism for Foundation Models

Foundation models have impressive performance and generalization capabilities across a wide range of applications. The increasing size of the models introduces great challenges for the training. Tensor parallelism is a critical technique that is currently used in almost all foundation model training and has a significant impact on overall training performance. However, current tensor parallelism in machine learning frameworks misses optimization opportunities in fitting various interconnection topologies. In this work, we present ATP, an adaptive tensor parallelism framework for foundation models, which can automatically select the optimal parallel strategy on different interconnections. We propose column- and row-first tensor parallelism based on 2D device meshes and construct a search space. Combined with the hierarchical communication matrix, ATP can identify the optimal strategy in the search space. We also propose chunk-based overlapping to reduce communication overhead. Our evaluations show ATP consistently outperforms the state-of-the-art approaches for various model sizes and interconnects, achieving end-to-end training performance improvements of up to 37-64% on specific interconnects. Based on our theoretical model, the communication overhead of ATP decreases with scaling, indicating a qualitative leap forward

RHCO: A Relation-aware Heterogeneous Graph Neural Network with Contrastive Learning for Large-scale Graphs

Heterogeneous graph neural networks (HGNNs) have been widely applied in heterogeneous information network tasks, while most HGNNs suffer from poor scalability or weak representation when they are applied to large-scale heterogeneous graphs. To address these problems, we propose a novel Relation-aware Heterogeneous Graph Neural Network with Contrastive Learning (RHCO) for large-scale heterogeneous graph representation learning. Unlike traditional heterogeneous graph neural networks, we adopt the contrastive learning mechanism to deal with the complex heterogeneity of large-scale heterogeneous graphs. We first learn relation-aware node embeddings under the network schema view. Then we propose a novel positive sample selection strategy to choose meaningful positive samples. After learning node embeddings under the positive sample graph view, we perform a cross-view contrastive learning to obtain the final node representations. Moreover, we adopt the label smoothing technique to boost the performance of RHCO. Extensive experiments on three large-scale academic heterogeneous graph datasets show that RHCO achieves best performance over the state-of-the-art models

Clinicopathologic Characteristics and Prognosis of Oligodendroglioma with IDH Mutation and 1p/19q Codeletion

Objective To analyze the clinicopathological characteristics and prognosis of oligodendroglioma with IDH mutation and 1p/19q codeletion. Methods We collected the data of 54 oligodendroglioma patients with IDH mutation and 1p/19q codeletion.The patients'clinicopathological data, including age, histological grade, and tumor site, were analyzed for the effects on progression-free and overall survival. Results Among the 54 patients, 46 cases were with tumor sites in one lobe, and eight cases involved tumor sites in more than two lobes.A total of 12 and 42 cases had WHO grades 2 and 3 oligodendroglioma, respectively.Detection by fluorescence in situ hybridization showed 1p/19q co-deletion in all cases.Immunohistochemical tests revealed diffuse and strong positive results for Olig2.All glial fibrillary acidic proteins were positive.p53 was strongly positive in six cases.ATRX was expressed in all 48 cases.Ki-67 proliferation index ranged from 5% to 60%.Sanger sequencing showed that all 54 cases had IDH gene mutations (40 cases were IDH1 mutations, and 14 were IDH2 mutations), and 33 cases had telomerase reverse transcriptase promoter mutations.Relapse and metastasis occurred in 16 patients during treatment.Univariate analysis indicated that the postoperative recurrence and metastasis interval of more than two years can prolong the progression-free and overall survival of patients.All 54 patients had a mean progression-free survival of 33.5 months and the mean overall survival of 40.7 months. Conclusion For oligodendroglioma with IDH mutation and 1p/19q codeletion, precision chemoradiotherapy after surgery can reduce the risk of progression, and the postoperative recurrence and metastasis interval is associated with the prognosis

Investigation of hearing loss in elderly vertigo and dizziness patients in the past 10 years

BackgroundVertigo and hearing loss are both prevalent in the elderly. This study retrospectively analyzed hearing test results from elderly patients experiencing vertigo and dizziness at ENT outpatient over a 10-year period, in order to study the patterns of hearing loss in this patient population.MethodsNine thousand three hundred eighty four patients over 50 years old underwent retrospective collection and screening of outpatient diagnosis, pure tone audiometry, acoustic immittance measurement (tympanogram) and auditory brainstem response (ABR) test. The patient's audiograms are divided into 7 subtypes according to a set of fixed criteria. Meanwhile, K-Means clustering analysis method was used to classify the audiogram.ResultsThe Jerger classification of tympanogram in elderly patients with vertigo and dizziness showed the majority falling under type A. The leading audiogram shapes were flat (27.81% in right ear and 26.89% in left ear), high-frequency gently sloping (25.97% in right ear and 27.34% in left ear), and high-frequency steeply sloping (21.60% in right ear and 22.53% in left ear). Meniere's disease (MD; 30.87%), benign recurrent vertigo (BRV; 19.07%), and benign paroxysmal positional vertigo (BPPV; 15.66%) were the most common etiologies in elderly vestibular diseases. We observed statistically significant differences in hearing thresholds among these vestibular diseases (P < 0.001). K-Means clustering analysis suggested that the optimal number of clusters was three, with sample sizes for the three clusters being 2,747, 2,413, and 4,139, respectively. The ANOVA statistical results of each characteristic value showed P < 0.001.ConclusionThe elderly patients often have mild to moderate hearing loss as a concomitant symptom with vertigo. Female patients have better hearing thresholds than males. The dominant audiometric shapes in this patient population were flat, high-frequency gently sloping, and high-frequency steeply sloping according to a set of fixed criteria. This study highlights the need for tailored strategies in managing hearing loss in elderly patients with vertigo and dizziness

Increased expression of programmed death ligand 1 (PD-L1) in human pituitary tumors

PURPOSE: Subsets of pituitary tumors exhibit an aggressive clinical courses and recur despite surgery, radiation, and chemotherapy. Because modulation of the immune response through inhibition of T-cell checkpoints has led to durable clinical responses in multiple malignancies, we explored whether pituitary adenomas express immune-related biomarkers that could suggest suitability for immunotherapy. Specifically, programmed death ligand 1 (PD-L1) has emerged as a potential biomarker whose expression may portend more favorable responses to immune checkpoint blockade therapies. We thus investigated the expression of PD-L1 in pituitary adenomas. METHODS: PD-L1 RNA and protein expression were evaluated in 48 pituitary tumors, including functioning and non-functioning adenomas as well as atypical and recurrent tumors. Tumor infiltrating lymphocyte populations were also assessed by immunohistochemistry. RESULTS: Pituitary tumors express variable levels of PD-L1 transcript and protein. PD-L1 RNA and protein expression were significantly increased in functioning (growth hormone and prolactin-expressing) pituitary adenomas compared to non-functioning (null cell and silent gonadotroph) adenomas. Moreover, primary pituitary adenomas harbored higher levels of PD-L1 mRNA compared to recurrent tumors. Tumor infiltrating lymphocytes were observed in all pituitary tumors and were positively correlated with increased PD-L1 expression, particularly in the functional subtypes. CONCLUSIONS: Human pituitary adenomas harbor PD-L1 across subtypes, with significantly higher expression in functioning adenomas compared to non-functioning adenomas. This expression is accompanied by the presence of tumor infiltrating lymphocytes. These findings suggest the existence of an immune response to pituitary tumors and raise the possibility of considering checkpoint blockade immunotherapy in cases refractory to conventional management

Delineating the molecular landscape of different histopathological growth patterns in colorectal cancer liver metastases

BackgroundHistopathological growth patterns (HGPs) have shown important prognostic values for patients with colorectal cancer liver metastases, but the potential molecular mechanisms remain largely unknown.MethodsWe performed an exploratory analysis by conducting the RNA sequencing of primary colorectal lesions, colorectal liver metastatic lesions and normal liver tissues.FindingsWe found that desmoplastic HGPs of the metastatic lesions were significantly enriched in EMT, angiogenesis, stroma, and immune signaling pathways, while replacement HGPs were enriched in metabolism, cell cycle, and DNA damage repair pathways. With the exception of immune-related genes, the differentially expressed genes of the two HGPs from colorectal liver metastases were mostly inherited from the primary tumor. Moreover, normal liver tissue in the desmoplastic HGP subgroup was markedly enriched in the fibrinous inflammation pathway.ConclusionsWe surmised that HGPs are observable morphological changes resulting from the regulation of molecular expressions, which is the combined effect of the heterogeneity and remodeling of primary tumors seeds and liver soils

Transboundary Water Cooperation in the Post-Cold War Era: Spatial Patterns and the Role of Proximity

Transboundary water cooperation (TWC) is an important theme of international cooperation. We conducted macro-level research on TWC from the perspective of inter-country relations and constructed a theoretical framework in which multidimensional proximity influences the formation of global TWC. We explained how multidimensional proximity and the constituent elements comprehensively influence the cooperative willingness and ability of actors, which directly drive the generation of global TWC. During the empirical research phase, we constructed the TWC frequency and intensity networks based on historical TWC events data from 1992 to 2013. By using social network analysis and QAP regression analysis, the spatial structure and proximity effect of water cooperation linkages are examined. It can be found that: (1) the reconstruction of territorial space on the eve of the end of the Cold War led to the peak of water cooperation events in 1992. The overall scale of events in the Post-Cold War era was relatively high and fluctuated steadily. (2) Water cooperation linkages have distinct spatial heterogeneity and are concentrated in the Eurasian and the African continents. Water cooperation is sensitive to geographical distance, and high-intensity water cooperation linkages exist in only a few areas. (3) China, Egypt, Germany, the United States, and Russia have prominent positions in the network. The United States, Japan, and other extra-regional powers actively participated in TWC in the Eastern Hemisphere. (4) The regression results show that geographical, economic, organizational, and colonial proximity significantly affect the intensity of water cooperation among countries