Tree Cross Attention

Cross Attention is a popular method for retrieving information from a set of context tokens for making predictions. At inference time, for each prediction, Cross Attention scans the full set of $\mathcal{O}(N)$ tokens. In practice, however, often only a small subset of tokens are required for good performance. Methods such as Perceiver IO are cheap at inference as they distill the information to a smaller-sized set of latent tokens $L < N$ on which cross attention is then applied, resulting in only $\mathcal{O}(L)$ complexity. However, in practice, as the number of input tokens and the amount of information to distill increases, the number of latent tokens needed also increases significantly. In this work, we propose Tree Cross Attention (TCA) - a module based on Cross Attention that only retrieves information from a logarithmic $\mathcal{O}(\log(N))$ number of tokens for performing inference. TCA organizes the data in a tree structure and performs a tree search at inference time to retrieve the relevant tokens for prediction. Leveraging TCA, we introduce ReTreever, a flexible architecture for token-efficient inference. We show empirically that Tree Cross Attention (TCA) performs comparable to Cross Attention across various classification and uncertainty regression tasks while being significantly more token-efficient. Furthermore, we compare ReTreever against Perceiver IO, showing significant gains while using the same number of tokens for inference.Comment: Accepted by ICLR 202

Constant Memory Attention Block

Modern foundation model architectures rely on attention mechanisms to effectively capture context. However, these methods require linear or quadratic memory in terms of the number of inputs/datapoints, limiting their applicability in low-compute domains. In this work, we propose Constant Memory Attention Block (CMAB), a novel general-purpose attention block that computes its output in constant memory and performs updates in constant computation. Highlighting CMABs efficacy, we introduce methods for Neural Processes and Temporal Point Processes. Empirically, we show our proposed methods achieve results competitive with state-of-the-art while being significantly more memory efficient.Comment: Workshop version of arXiv:2305.1456

Bayesian topology learning and noise removal from network data

Learning the topology of a graph from available data is of great interest in many emerging applications. Some examples are social networks, internet of things networks (intelligent IoT and industrial IoT), biological connection networks, sensor networks and traffic network patterns. In this paper, a graph topology inference approach is proposed to learn the underlying graph structure from a given set of noisy multi-variate observations, which are modeled as graph signals generated from a Gaussian Markov Random Field (GMRF) process. A factor analysis model is applied to represent the graph signals in a latent space where the basis is related to the underlying graph structure. An optimal graph filter is also developed to recover the graph signals from noisy observations. In the final step, an optimization problem is proposed to learn the underlying graph topology from the recovered signals. Moreover, a fast algorithm employing the proximal point method has been proposed to solve the problem efficiently. Experimental results employing both synthetic and real data show the effectiveness of the proposed method in recovering the signals and inferring the underlying graph

17β-HSD13 has sex-based differential expression in Hepatitis C virus-induced cirrhosis and hepatocellular carcinoma

Background: Sex-based differences are observed in chronic hepatitis C virus (HCV) infections leading to cirrhosis and hepatocellular carcinoma (HCC). We previously showed that liver estrogen receptor (ER-) mediated sex-based differences exist in cirrhosis and HCC. Liver ER-binding may lead to protective effects in pre-menopausal women. This study aimed to determine sex-based differential role of 17βHSD13 in development of cirrhosis and HCC. We hypothesized that chronic HCV infection leads to dysregulated 17β-HSD13 in male cirrhosis and progression to HCC.Methods: 65 (normal, cirrhosis, HCC) liver tissues were obtained from NIH Liver Tissue Bank. DIA proteomics mapped 4445 proteins, including 17β-HSD13. Clinical correlation with bilirubin, AST, ALP, and creatinine was determined (spearman’s). Immunohistochemistry validated 17β-HSD13 protein expression in tissues.Results: 17β-HSD13 had significantly lower expression in male cirrhosis group than females (P<0.05). In contrast, 17β-HSD13 expression in normal males was significantly greater than normal females (P<0.05). In HCC group, the expression in males was down-regulated compared to HCC females (P<0.05). Bilirubin values showed negative correlation with 17β-HSD13 expression (P<0.05) between cirrhosis and HCC (males alone and combined sex data).Conclusions: Low 17β-HSD13 levels may predict worse disease in males with cirrhosis or HCC serving as disease biomarker. This novel report shows sex-based differences in 17β-HSD13 during HCV-induced cirrhosis development