15 research outputs found
Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop
Large language models (LLM) have become state of the art in many benchmarks
and conversational LLM applications like ChatGPT are now widely used by the
public. Those LLMs can be used to generate large amounts of content which is
posted on the internet to various platforms. As LLMs are trained on datasets
usually collected from the internet, this LLM-generated content might be used
to train the next generation of LLMs. Therefore, a self-consuming training loop
emerges in which new LLM generations are trained on the output from the
previous generations. We empirically study this self-consuming training loop
using a novel dataset to analytically and accurately measure quality and
diversity of generated outputs. We find that this self-consuming training loop
initially improves both quality and diversity. However, after a few generations
the output inevitably degenerates in diversity. We find that the rate of
degeneration depends on the proportion of real and generated data
MTGP: Combining Metamorphic Testing and Genetic Programming
Genetic programming is an evolutionary approach known for its performance in
program synthesis. However, it is not yet mature enough for a practical use in
real-world software development, since usually many training cases are required
to generate programs that generalize to unseen test cases. As in practice, the
training cases have to be expensively hand-labeled by the user, we need an
approach to check the program behavior with a lower number of training cases.
Metamorphic testing needs no labeled input/output examples. Instead, the
program is executed multiple times, first on a given (randomly generated)
input, followed by related inputs to check whether certain user-defined
relations between the observed outputs hold. In this work, we suggest MTGP,
which combines metamorphic testing and genetic programming and study its
performance and the generalizability of the generated programs. Further, we
analyze how the generalizability depends on the number of given labeled
training cases. We find that using metamorphic testing combined with labeled
training cases leads to a higher generalization rate than the use of labeled
training cases alone in almost all studied configurations. Consequently, we
recommend researchers to use metamorphic testing in their systems if the
labeling of the training data is expensive
Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Content
This paper examines how individuals perceive the credibility of content
originating from human authors versus content generated by large language
models, like the GPT language model family that powers ChatGPT, in different
user interface versions. Surprisingly, our results demonstrate that regardless
of the user interface presentation, participants tend to attribute similar
levels of credibility. While participants also do not report any different
perceptions of competence and trustworthiness between human and AI-generated
content, they rate AI-generated content as being clearer and more engaging. The
findings from this study serve as a call for a more discerning approach to
evaluating information sources, encouraging users to exercise caution and
critical thinking when engaging with content generated by AI systems
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
To support software developers in finding and fixing software bugs, several
automated program repair techniques have been introduced. Given a test suite,
standard methods usually either synthesize a repair, or navigate a search space
of software edits to find test-suite passing variants. Recent program repair
methods are based on deep learning approaches. One of these novel methods,
which is not primarily intended for automated program repair, but is still
suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is
so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard
bug fixing benchmark set, QuixBugs, and compare the performance with the
results of several other approaches reported in the literature. We find that
ChatGPT's bug fixing performance is competitive to the common deep learning
approaches CoCoNut and Codex and notably better than the results reported for
the standard program repair approaches. In contrast to previous approaches,
ChatGPT offers a dialogue system through which further information, e.g., the
expected output for a certain input or an observed error message, can be
entered. By providing such hints to ChatGPT, its success rate can be further
increased, fixing 31 out of 40 bugs, outperforming state-of-the-art
Enhancing Genetic Improvement Mutations Using Large Language Models
Large language models (LLMs) have been successfully applied to software
engineering tasks, including program repair. However, their application in
search-based techniques such as Genetic Improvement (GI) is still largely
unexplored. In this paper, we evaluate the use of LLMs as mutation operators
for GI to improve the search process. We expand the Gin Java GI toolkit to call
OpenAI's API to generate edits for the JCodec tool. We randomly sample the
space of edits using 5 different edit types. We find that the number of patches
passing unit tests is up to 75% higher with LLM-based edits than with standard
Insert edits. Further, we observe that the patches found with LLMs are
generally less diverse compared to standard edits. We ran GI with local search
to find runtime improvements. Although many improving patches are found by
LLM-enhanced GI, the best improving patch was found by standard GI.Comment: Accepted for publication at the Symposium on Search-Based Software
Engineering (SSBSE) 202
The Randomness of Input Data Spaces is an A Priori Predictor for Generalization
Over-parameterized models can perfectly learn various types of data
distributions, however, generalization error is usually lower for real data in
comparison to artificial data. This suggests that the properties of data
distributions have an impact on generalization capability. This work focuses on
the search space defined by the input data and assumes that the correlation
between labels of neighboring input values influences generalization. If
correlation is low, the randomness of the input data space is high leading to
high generalization error. We suggest to measure the randomness of an input
data space using Maurer's universal. Results for synthetic classification tasks
and common image classification benchmarks (MNIST, CIFAR10, and Microsoft's
cats vs. dogs data set) find a high correlation between the randomness of input
data spaces and the generalization error of deep neural networks for binary
classification problems
Using Knowledge of Human-Generated Code to Bias the Search in Program Synthesis with Grammatical Evolution
Enhancing Genetic Improvement Mutations Using Large Language Models
Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI
Enhancing Genetic Improvement Mutations Using Large Language Models
Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI