Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion

Pretrained code language models have enabled great progress towards program synthesis. However, common approaches only consider in-file local context and thus miss information and constraints imposed by other parts of the codebase and its external dependencies. Existing code completion benchmarks also lack such context. To resolve these restrictions we curate a new dataset of permissively licensed Python packages that includes full projects and their dependencies and provide tools to extract non-local information with the help of program analyzers. We then focus on the task of function call argument completion which requires predicting the arguments to function calls. We show that existing code completion models do not yield good results on our completion task. To better solve this task, we query a program analyzer for information relevant to a given function call, and consider ways to provide the analyzer results to different code completion models during inference and training. Our experiments show that providing access to the function implementation and function usages greatly improves the argument completion performance. Our ablation study provides further insights on how different types of information available from the program analyzer and different ways of incorporating the information affect the model performance.Comment: 12 pages. Accepted to AAAI 202

Parameter and Data Efficient Continual Pre-training for Robustness to Dialectal Variance in Arabic

The use of multilingual language models for tasks in low and high-resource languages has been a success story in deep learning. In recent times, Arabic has been receiving widespread attention on account of its dialectal variance. While prior research studies have tried to adapt these multilingual models for dialectal variants of Arabic, it still remains a challenging problem owing to the lack of sufficient monolingual dialectal data and parallel translation data of such dialectal variants. It remains an open problem on whether the limited dialectical data can be used to improve the models trained in Arabic on its dialectal variants. First, we show that multilingual-BERT (mBERT) incrementally pretrained on Arabic monolingual data takes less training time and yields comparable accuracy when compared to our custom monolingual Arabic model and beat existing models (by an avg metric of +$6.41$). We then explore two continual pre-training methods -- (1) using small amounts of dialectical data for continual finetuning and (2) parallel Arabic to English data and a Translation Language Modeling loss function. We show that both approaches help improve performance on dialectal classification tasks ($+4.64$ avg. gain) when used on monolingual models

Large Language Models of Code Fail at Completing Code with Potential Bugs

Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence of bugs in the code context for generation, which are inevitable in software development. Therefore, we introduce and study the buggy-code completion problem, inspired by the realistic scenario of real-time code suggestion where the code context contains potential bugs -- anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with synthetic bugs derived from semantics-altering operator changes (buggy-HumanEval) and one with realistic bugs derived from user submissions to coding problems (buggy-FixEval). We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs. For instance, the passing rates of CodeGen-2B-mono on test cases of buggy-HumanEval drop more than 50% given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains a large gap in post-mitigation performance.Comment: 25 page

HYTREL: Hypergraph-enhanced Tabular Data Representation Learning

Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks. However, many of these models do not take into account the row/column permutation invariances, hierarchical structure, etc. that exist in tabular data. To alleviate these limitations, we propose HYTREL, a tabular language model, that captures the permutation invariances and three more structural properties of tabular data by using hypergraphs - where the table cells make up the nodes and the cells occurring jointly together in each row, column, and the entire table are used to form three different types of hyperedges. We show that HYTREL is maximally invariant under certain conditions for tabular data, i.e., two tables obtain the same representations via HYTREL iff the two tables are identical up to permutations. Our empirical results demonstrate that HYTREL consistently outperforms other competitive baselines on four downstream tasks with minimal pretraining, illustrating the advantages of incorporating the inductive biases associated with tabular data into the representations. Finally, our qualitative analyses showcase that HYTREL can assimilate the table structures to generate robust representations for the cells, rows, columns, and the entire table.Comment: NeurIPS 2023 (spotlight

Non-Standard Errors

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: Non-standard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for better reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants

CrowdRisk: exploring crowdsourcing of risk information

This paper describes the outcomes of a preliminary study into the design of a mobile app to crowdsource information related to “risk”. For the purpose of this study the notion of risk is defined broadly; however, we predominantly focus on the personal, subjective perception of risk. The study involved building a prototypical mobile app to crowdsource risk and exploring the use of the app as part of an expert workshop. Outcomes show challenges and opportunities with regards to the categorisation of results, the motivation of users, and interaction design of the prototype. The study provides value by giving an initial insight into this design space

Targets of the Tal1 transcription factor in erythrocytes: E2 ubiquitin conjugase regulation by Tal1

The Tal1 transcription factor is essential for the development of the hematopoietic system and plays a role during definitive erythropoiesis in the adult. Despite the importance of Tal1 in erythropoiesis only a small number of erythroid differentiation target genes are known. A chromatin precipitation and cloning approach was established to uncover novel Tal1 target genes in erythropoiesis. The BirA-tag/BirA-ligase biotinylation system in combination with streptavidine chromatin precipitation (Strep-CP) was used to co-precipitate genomic DNA bound to Tal1. Tal1 was found to bind in the vicinity of 31 genes including the E2-ubiquitin conjugase UBE2H gene. Binding of Tal1 to UBE2H was confirmed by chromatin immuno-precipitation. UBE2H expression is increased during erythroid differentiation of hCD34+ cells. Tal1 expression activated UBE2H expression whereas Tal1 knock-down reduced UBE2H expression and ubiquitin transfer activity. This study identifies parts of the ubiquitinylation machinery as a cellular target down-stream of the transcription factor Tal1 and provides novel insights into Tal1 regulated erythropoiesis