What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?

Pre-trained models of source code have gained widespread popularity in many code intelligence tasks. Recently, with the scaling of the model and corpus size, large language models have shown the ability of in-context learning (ICL). ICL employs task instructions and a few examples as demonstrations, and then inputs the demonstrations to the language models for making predictions. This new learning paradigm is training-free and has shown impressive performance in various natural language processing and code intelligence tasks. However, the performance of ICL heavily relies on the quality of demonstrations, e.g., the selected examples. It is important to systematically investigate how to construct a good demonstration for code-related tasks. In this paper, we empirically explore the impact of three key factors on the performance of ICL in code intelligence tasks: the selection, order, and number of demonstration examples. We conduct extensive experiments on three code intelligence tasks including code summarization, bug fixing, and program synthesis. Our experimental results demonstrate that all the above three factors dramatically impact the performance of ICL in code intelligence tasks. Additionally, we summarize our findings and provide takeaway suggestions on how to construct effective demonstrations, taking into account these three perspectives. We also show that a carefully-designed demonstration based on our findings can lead to substantial improvements over widely-used demonstration construction methods, e.g., improving BLEU-4, EM, and EM by at least 9.90%, 175.96%, and 50.81% on code summarization, bug fixing, and program synthesis, respectivelyComment: This paper is accepted by ASE 202

Scaling Law of Large Sequential Recommendation Models

Scaling of neural networks has recently shown great potential to improve the model capacity in various fields. Specifically, model performance has a power-law relationship with model size or data size, which provides important guidance for the development of large-scale models. However, there is still limited understanding on the scaling effect of user behavior models in recommender systems, where the unique data characteristics (e.g. data scarcity and sparsity) pose new challenges to explore the scaling effect in recommendation tasks. In this work, we focus on investigating the scaling laws in large sequential recommendation models. Specially, we consider a pure ID-based task formulation, where the interaction history of a user is formatted as a chronological sequence of item IDs. We don't incorporate any side information (e.g. item text), because we would like to explore how scaling law holds from the perspective of user behavior. With specially improved strategies, we scale up the model size to 0.8B parameters, making it feasible to explore the scaling effect in a diverse range of model sizes. As the major findings, we empirically show that scaling law still holds for these trained models, even in data-constrained scenarios. We then fit the curve for scaling law, and successfully predict the test loss of the two largest tested model scales. Furthermore, we examine the performance advantage of scaling effect on five challenging recommendation tasks, considering the unique issues (e.g. cold start, robustness, long-term preference) in recommender systems. We find that scaling up the model size can greatly boost the performance on these challenging tasks, which again verifies the benefits of large recommendation models

Segmentation of kidney lesions with attention model based on Deeplab

We participate this challenge by developing a hierarchical framework. We build the model from two fully convolutional networks: (1) a simple Unet model to normalize the input iamges, (2) a segmentaion network which is an attention model based on Deeplab model. Two models are connected in tandem and trained end-to-end. To ensure a better results, we use the preprocess method proposed by nnUnet in our experiments

Toll-like receptor 9 negatively regulates pancreatic islet beta cell growth and function in a mouse model of type 1 diabetes

Aims/hypothesis Innate immune effectors interact with the environment to contribute to the pathogenesis of the autoimmune disease, type 1 diabetes. Although recent studies have suggested that innate immune Toll-like receptors (TLRs) are involved in tissue development, little is known about the role of TLRs in tissue development, compared with autoimmunity. We aimed to fill the knowledge gap by investigating the role of TLR9 in the development and function of islet beta cells in type 1 diabetes, using NOD mice. Methods We generated Tlr9−/− NOD mice and examined them for type 1 diabetes development and beta cell function, including insulin secretion and glucose tolerance. We assessed islet and beta cell number and characterised CD140a expression on beta cells by flow cytometry. We also tested beta cell function in Tlr9−/− C57BL/6 mice. Finally, we used TLR9 antagonists to block TLR9 signalling in wild-type NOD mice to verify the role of TLR9 in beta cell development and function. Results TLR9 deficiency promoted pancreatic islet development and beta cell differentiation, leading to enhanced glucose tolerance, improved insulin sensitivity and enhanced first-phase insulin secretory response. This was, in part, mediated by upregulation of CD140a (also known as platelet-derived growth factor receptor-α [PDGFRα]). In the absence of TLR9, induced by either genetic targeting or treatment with TLR9 antagonists, which had similar effects on ontogenesis and function of beta cells, NOD mice were protected from diabetes. Conclusions/interpretation Our study links TLR9 and the CD140a pathway in regulating islet beta cell development and function and indicates a potential therapeutic target for diabetes prevention and/or treatment