67 research outputs found
On-the-Fly Adaptation of Source Code Models using Meta-Learning
The ability to adapt to unseen, local contexts is an important challenge that
successful models of source code must overcome. One of the most popular
approaches for the adaptation of such models is dynamic evaluation. With
dynamic evaluation, when running a model on an unseen file, the model is
updated immediately after having observed each token in that file. In this
work, we propose instead to frame the problem of context adaptation as a
meta-learning problem. We aim to train a base source code model that is best
able to learn from information in a file to deliver improved predictions of
missing tokens. Unlike dynamic evaluation, this formulation allows us to select
more targeted information (support tokens) for adaptation, that is both before
and after a target hole in a file. We consider an evaluation setting that we
call line-level maintenance, designed to reflect the downstream task of code
auto-completion in an IDE. Leveraging recent developments in meta-learning such
as first-order MAML and Reptile, we demonstrate improved performance in
experiments on a large scale Java GitHub corpus, compared to other adaptation
baselines including dynamic evaluation. Moreover, our analysis shows that,
compared to a non-adaptive baseline, our approach improves performance on
identifiers and literals by 44\% and 15\%, respectively.Comment: This paper has been withdrawn because we found a bug in the FOMAML
implementation that invalidates some of the key claims in the pape
Repository-Level Prompt Generation for Large Language Models of Code
With the success of large language models (LLMs) of code and their use as
code assistants (e.g. Codex used in GitHub Copilot), techniques for introducing
domain-specific knowledge in the prompt design process become important. In
this work, we propose a framework called Repo-Level Prompt Generator that
learns to generate example-specific prompts using prompt proposals. The prompt
proposals take context from the entire repository, thereby incorporating both
the structure of the repository and the context from other relevant files (e.g.
imports, parent class files). Our technique doesn't require any access to the
weights of the LLM, making it applicable in cases where we only have black-box
access to the LLM. We conduct experiments on the task of single-line
code-autocompletion using code repositories taken from Google Code archives. We
demonstrate that an oracle constructed from our prompt proposals gives a
remarkably high relative improvement of 36% over Codex, showing the quality of
these proposals. Further, we show that when we train a model to predict a
prompt proposal, we can achieve significant performance gains over Codex and
other baselines. The code for our work can be found at:
\url{https://github.com/shrivastavadisha/repo_level_prompt_generation}
Consensus Message Passing for Layered Graphical Models
Generative models provide a powerful framework for probabilistic reasoning.
However, in many domains their use has been hampered by the practical
difficulties of inference. This is particularly the case in computer vision,
where models of the imaging process tend to be large, loopy and layered. For
this reason bottom-up conditional models have traditionally dominated in such
domains. We find that widely-used, general-purpose message passing inference
algorithms such as Expectation Propagation (EP) and Variational Message Passing
(VMP) fail on the simplest of vision models. With these models in mind, we
introduce a modification to message passing that learns to exploit their
layered structure by passing 'consensus' messages that guide inference towards
good solutions. Experiments on a variety of problems show that the proposed
technique leads to significantly more accurate inference results, not only when
compared to standard EP and VMP, but also when compared to competitive
bottom-up conditional models.Comment: Appearing in Proceedings of the 18th International Conference on
Artificial Intelligence and Statistics (AISTATS) 201
- …